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United  States 

General  Accounting  Office 

Washington,  D.C.  20548 


GAO 

Accounting  and  Information 
Management  Division 

B-275670 

October  20, 1997 

The  Honorable  John  Glenn 
Ranking  Minority  Member 
Committee  on  Governmental  Affairs 
United  States  Senate 

Dear  Senator  Glenn: 

As  you  know,  in  1989,  the  Department  of  Defense  started  its  Corporate  Information 
Management  (cm)  initiative  in  an  effort  to  save  billions  of  dollars  by  streamlining  operations 
and  deploying  standard  information  systems  to  support  common  business  operations.  A  key 
part  of  this  initiative  is  Defense’s  migration  effort,  which  involves  replacing  its  functionally 
duplicative  and  inefficient  automated  information  systems  with  the  Department’s  best  existing 
information  systems.  Concerned  that  the  biUions  of  dollars  projected  to  be  spent  on  cm-related 
technology  efforts  are  at  high  risk,  you  asked  that  we  provide  information  on  the  status  and 
progress  of  Defense’s  migration  effort  and  assess  whether  Defense  has  effective  controls  in 
place  to  manage  and  oversee  the  initiative.  This  report  also  highlights  the  implications  of  our 
findings  for  Defense  as  it  responds  to  investment  management  requirements  of  the 
Clinger-Cohen  Act  of  1996. 

We  are  sending  copies  of  this  report  to  the  Chairman  of  the  Senate  Committee  on  Governmental 
Affairs  and  to  the  Chairmen  and  Ranking  Minority  Members  of  the  Senate  Conunittee  on  Armed 
Services;  Subcommittee  on  Defense,  Senate  Committee  on  Appropriations;  House  Committee 
on  National  Security;  Subcommittee  on  National  Security,  House  Committee  on  Appropriations; 
and  Senate  and  House  Committees  on  the  Budget;  the  Secretary  of  Defense;  the  Acting 
Assistant  Secretary  of  Defense  for  Command,  Control,  Communications  and  Intelligence;  the 
Acting  Under  Secretary  of  Defense  (Comptroller);  and  the  Director,  Office  of  Management  and 
Budget.  Copies  will  also  be  made  available  to  others  upon  request. 

If  you  have  any  questions  about  this  report,  please  call  Mickey  McDermott,  Assistant  Director, 
at  (202)  512-6240.  Other  major  contributors  to  this  report  are  listed  in  appendix  V. 

Sincerely  yours. 


Jack  L.  Brock,  Jr. 

Director,  Defense  Iitformation  and 
Financial  Management  Systems 


Executive  Summary 


Purpose 


Background 


In  1989,  the  Department  of  Defense  started  its  Corporate  Information 
Management  (cim)  initiative  in  an  effort  to  save  billions  of  dollars  by 
streamlining  operations  and  deploying  standard  information  systems  to 
support  common  business  operations.  However,  8  years  after  beginning 
CIM  and  making  substantial  investments.  Defense  has  not  met  its  savings 
goal  because  it  has  not  fully  implemented  sound  management  practices  to 
carry  out  this  initiative,  gao  placed  the  effort  on  its  high-risk  list  in 
February  1995,  in  part  because  it  found  that  ciM-related  technology 
investments,  which  are  expected  to  total  billions  of  dollars  each  year,  are 
vulnerable  to  waste  and  mismanagement. 

A  key  part  of  cim  is  Defense’s  migration  effort,  which  involves  replacing  its 
functionally  duplicative  and  inefficient  automated  information  systems 
with  the  best  existing  systems.  Defense  believes  that  migration  can  cut 
costs  associated  with  developing  and  maintaining  disparate  systems 
supporting  the  same  functions.  It  also  beUeves  that  if  done  properly, 
migration  can  help  standardize  business  processes.  Concerned  that 
bUhons  of  dollars  projected  to  be  spent  on  ciM-related  technology  efforts 
are  at  high  risk,  the  Ranking  Minority  Member  of  the  Senate  Committee  on 
Governmental  Affairs  asked  gao  to  (1)  provide  information  on  the  number 
and  cost  of  systems  designated  for  migration,  the  number  of  legacy 
systems  already  terminated  or  scheduled  for  termination,  and  savings 
resulting  from  terminations  of  legacy  systems  and  (2)  determine  whether 
Defense’s  management  control  and  oversight  processes  for  migration 
systems  are  ensuring  that  the  investments  are  economically  sound  and  in 
compliance  with  its  technical  and  data  standards. 


When  it  initiated  cim.  Defense  believed  that  the  thousands  of  automated 
information  systems  supporting  its  business  operations — ^which  include 
such  functions  as  logistics,  communications,  personnel,  health  affairs,  and 
finance — were  redundant  and  inefficient.  These  operations  were 
traditionally  carried  out  in  isolation  by  the  individual  military  services  and 
Defense  agencies.  As  such,  the  information  systems  supporting  these 
discrete  operations  were  developed  independently  even  though  they  may 
well  have  served  similar  purposes. 

dod’s  CIM  initiative  included  several  aspects:  (1)  corporate  pohcy/planning, 
(2)  process  and  data  modeling,  (3)  process  improvement,  (4)  performance 
measurement,  (5)  standard  information  systems,  and  (6)  computing  and 
communications  infrastructure.  As  cim  was  implemented.  Defense 
emphasized  two  ways  of  achieving  process  improvements  and  addressing 
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problems  associated  with  its  disparate  and  stovepiped  information 
technology  environment:  (1)  reengineering  business  processes  first  and 
then  applying  technology  to  the  new  processes  and  (2)  selecting  the  best 
DOD  information  systems  from  pools  of  existing,  or  legacy,  systems  that 
provide  the  same  automated  support  services  and  eventually  replacing  the 
duplicative  systems  with  the  best  systems.  The  second  approach  is  known 
as  migration. 

In  October  1993,  Defense  embarked  on  an  “accelerated”  migration 
strategy,  which  placed  more  emphasis  on  the  second  improvement 
approach.  As  part  of  this  strategy,  it  asked  its  managers  to  select  migration 
systems  in  6  months  and  develop  and  deploy  them  departmentwide  in  the 
following  3  years.  Defense  believed  that  instilling  pressure  to  select  and 
deploy  migration  systems  would  reap  savings  much  quicker  than 
streamlining  or  reengineering  the  business  processes  and  acquiring 
systems  after  those  processes  were  reengineered. 

The  consequence  of  the  increased  emphasis  on  migration  was  that  the 
dramatic  gains  that  could  be  achieved  through  reengineering  would  be 
postponed.  This  impaired  the  chances  of  CM  achieving  its  objectives  for 
dramatic  improvements  and  cost  savings  in  several  respects.  First,  it  kept 
Defense  from  focusing  on  redesigning  core  business  processes,  which 
promised  order-of-magnitude  improvements.  Second,  it  increased  the  risk 
that  bad  business  processes  would  be  perpetuated.  Third,  it  would  make 
future  reengineering  efforts  more  difficult  by  entrenching  inefficient  and 
ineffective  work  processes.  As  a  result,  reengineering  never  took  hold  in 
many  of  dod’s  functional  areas.  The  migration  arm  of  cm,  however,  is  stiU 
active. 


Results  in  Brief 


From  fiscal  years  1995  through  2000,  Defense  plans  to  spend  at  least 
$18  billion  on  migration.  It  has  selected  363  migration  systems  and  has 
targeted  1,938  legacy  systems  for  potential  termination.  Despite  this 
substantial  investment.  Defense  did  not  adhere  to  decision-making  and 
oversight  processes  it  established  to  ensure  that  the  economical  and 
technical  risks  associated  with  migration  projects  have  been  mitigated. 

Defense’s  primary  corporate-level  control  mechanism  for  ensuring  that 
sound  management  and  development  practices  were  followed  for  each 
migration  investment  has  not  been  effective.  This  control  requires  that  all 
migration  system  selections  be  approved  by  the  Assistant  Secretary  of 
Defense  for  Command,  Control,  Communications  and  Intelligence 
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(asd  C3I).  Despite  this  requirement,  about  67  percent  of  migration 
selections  were  never  submitted  for  approval,  and  many  of  those  that  were 
submitted  were  approved  in  the  absence  of  critical  technical  and 
programmatic  support.  Consequently,  Defense  does  not  have  assurance 
that  the  migration  systems  being  developed  will  help  achieve  its 
technology  goals  and  that  sound  business  decisions  were  made  in 
selecting  the  systems. 

Because  this  control  mechanism  did  not  support  the  migration  effort,  gao 
also  assessed  whether  Defense’s  traditional  acquisition  oversight 
processes,  which  are  designed  to  help  ensure  that  individual  major 
information  system  investments  are  economically  and  technically  sormd, 
were  helping  to  ensure  that  the  risks  associated  with  migration  were 
mitigated.  However,  these  processes  also  did  not  support  the  migration 
effort  because  they  have  not  fully  ensured  that  economic  analyses  for 
migration  projects  are  prepared  and  reviewed  and  that  the  systems  comply 
with  technical  and  data  standards.  For  example,  economic  analyses  for  12 
of  43  major  migration  systems — for  which  Defense  has  invested  hundreds 
of  millions  of  dollars  in  total — have  not  yet  been  submitted  for 
independent  review.  Delaying  the  preparation  of  an  economic  analysis  to 
the  later  stages  of  development  defeats  the  purpose  of  the  analysis — to 
demonstrate  that  a  proposal  to  invest  in  a  new  system  is  valid  before  that 
investment  is  made. 

Had  there  been  more  rigorous  attention  to  established  oversight 
procedures  and  soimd  business  practices.  Defense  might  well  have 
avoided  the  migration  problems  gao  identified  in  previous  reviews,  which 
urmecessarily  cost  the  Department  hundreds  of  millions  of  dollars.  One 
functional  area,  for  example,  embarked  on  and  later  abandoned  a 
substantially  flawed  effort  to  develop  a  standard  suite  of  migration 
systems  for  materiel  management  after  spending  over  $700  million  without 
strong  oversight.  In  addition,  some  functional  areas  did  not  account  for 
various  categories  of  significant  costs  when  making  their  migration 
decisions  or  adequately  consider  alternatives  to  developing  systems 
in-house. 

In  implementing  the  migration  strategy,  Defense  did  not  ensure  that  it  had 
adequate  departmentwide  visibility  over  status,  costs,  and  progress.  As  a 
result,  it  could  not  provide  accurate  and  reliable  information,  as  requested, 
on  the  number  and  cost  of  systems  designated  for  migration,  and  the 
numbers  of  systems  terminated  and  scheduled  to  be  terminated. 
Furthermore,  the  Department  has  not  been  able  to  convincingly 
demonstrate  whether  the  migration  strategy  has  been  successful  or  not. 
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and  it  has  not  been  able  to  provide  the  Congress  with  accurate  and  reliable 
information  needed  for  decision-making  purposes. 

Defense  has  taken  a  positive  step  to  turn  around  its  information 
technology  investment  process  as  part  of  its  implementation  of  the 
Clinger-Cohen  Act  of  1996  (Division  E  of  Public  Law  104-106).  This 
legislation  requires  federal  agencies  to  have  processes  and  information  in 
place  to  help  ensure  that  information  technology  projects  (1)  are  being 
implemented  at  acceptable  costs,  within  reasonable  and  expected  time 
frames,  and  (2)  are  contributing  to  tangible,  observable  improvements  in 
mission  performance.  In  June  1997,  the  Secretary  of  Defense  outlined  his 
expectations  for  improvements  in  management  processes  and  information 
resovurces  related  to  information  technology.  These  expectations 
incorporate  some  of  the  most  important  elements  of  the  Clinger-Cohen  Act 
and  are  an  excellent  starting  point  for  bringing  meaningful  change  to  the 
current  information  technology  management  process.  Additionally,  the 
Department’s  Task  Force  on  Defense  Reform  is  currently  examining  the 
current  structure  of  the  Chief  Information  Officer  (ao)  position  to  ensure 
that  the  cio  can  devote  full  attention  to  reforming  information  resources 
management  within  the  Department. 

In  implementing  the  Clinger-Cohen  Act,  the  Department  faces  a  formidable 
challenge  in  successfully  implementing  real  change  across  an 
organizational  structure  that  has  clearly  defined  roles  and  responsibilities 
for  the  three  individual  services  to  execute  national  defense  policy  and 
objectives.  The  separation  of  budget  authority,  program  execution,  and 
functional  authority  have  all  contributed  to  an  environment  that  has 
fostered  stovepipe  systems  within  each  service  and  has  made 
departmentwide  oversight  difficult.  For  Clinger-Cohen  implementation  to 
make  a  difference,  any  new  processes  or  requirements  must  be 
successfully  accomplished  within  this  environment. 


Principal  Findings 


Management  Control 
Processes  Over  Individual 
Migration  Efforts  Have 
Broken  Down 


Defense’s  primary  corporate-level  control  mechanism  for  ensuring  that 
sound  management  and  development  practices  are  followed  for  each 
migration  investment  has  not  been  effective.  This  control  requires  the 
functional  area  manager,  kno-wn  as  the  Principal  Staff  Assistant^  (psa),  to 


^The  PSA  is  the  senior  executive-level  manager  who  is  responsible  for  the  management  of  a  defined 
function  or  functions  within  the  DOD. 
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submit  all  systems  selected  for  migration  for  approv2il  by  the  Assistant 
Secretary  of  Defense  for  Command,  Control,  Communications  and 
Intelligence.  This  approval  is  important  because  the  Assistant  Secretary,  as 
the  Department’s  Chief  Information  Officer,  needs  to  ensure  that  dod’s 
technology-related  goals  are  met. 

However,  this  control  mechanism  has  broken  down  in  two  respects.  First, 
even  though  all  system  selections  should  have  been  submitted  for  review 
and  approval  by  the  Assistant  Secretary,  245  of  the  363  selections  were 
never  submitted  for  this  oversight.  Second,  most  of  the  systems  that  were 
submitted  for  this  review  were  approved  in  the  absence  of  critical 
technical  and  programmatic  support.  Specifically,  about  19  percent  were 
approved  without  supporting  technical  justification  and  about  54  percent 
were  approved  without  documents  showing  that  the  functional  area  had 
evaluated  different  options  for  improving  a  business  area,  such  as 
reengineering.  Because  all  of  the  migration  selections  submitted  for  this 
oversight  have  been  approved — ^whether  adequately  supported  with 
technical  and  programmatic  justification  or  not — this  control  has 
essentially  become  meaningless. 

Because  the  oversight  from  the  asd  C3i  that  was  specifically  established 
for  migration  selections  has  been  marginal  in  trying  to  ensure  that  sound 
management  and  development  practices  are  followed  for  migration 
selection,  gao  assessed  whether  the  Department’s  traditional  major 
information  systems  acquisition  oversight  processes  were  doing  so  for  43 
of  the  major  migration  systems.^  Major  systems  eire  Defense’s  most 
expensive  and  critical  information  systems  and  can  cost  up  to  hundreds  of 
miUions  of  dollars  to  develop.  These  review  processes  are  designed  to 
assess  whether  projects  are  affordable  and  financial  and  operational  risks 
have  been  minimized. 

However,  these  processes  did  not  provide  sufficient  assurance  that  the 
systems  (1)  were  economically  justified  or  (2)  complied  with  Defense’s 
technical  and  data  standards — which  are  intended  to  help  pave  the  way 
toward  am  interoperable  systems  environment.  For  example,  economic 
analyses  or  updates  of  previously  prepared  economic  analyses  for  12  of 
the  43  systems  had  not  been  submitted  for  independent  review.  These  12 
systems  are  under  development  or  modernization  and  Defense  has 
invested  hundreds  of  millions  of  dollars  in  them.  In  addition,  even  though 
the  Department  recognizes  that  economic  analyses  play  a  critical  role  in 


-There  are  49  major  migration  systems;  however,  GAO’s  review  focused  on  the  43  major  migration 
systems  under  the  oversight  of  the  Major  Automated  Information  System  Review  Council.  The 
remaining  six  systems  are  under  the  oversight  of  the  Defense  Acquisition  Board. 
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assessing  whether  system  development  efforts  will  be  cost-effective  and 
beneficial,  it  has  not  yet  published  official  guidance  to  standardize  the 
methodology  and  analytic  techniques  for  preparing  economic  analyses. 

Gao’s  previous  reviews  of  migration  systems  identified  a  number  of 
problems  that  could  have  been  prevented  had  there  been  better  oversight 
by  the  Major  Automated  Information  System  Review  Council  (maisrc)  and 
the  ASD  C3L  For  example,  gao’s  review  of  the  materiel  management 
migration  strategy  showed  that  a  functional  area  was  able  to  embark  and 
spend  over  $700  million  pursuing  a  substantially  flawed  effort— which  was 
later  abandoned— without  rigorous  department-level  oversight.  In 
addition,  in  previous  reviews  of  migration  efforts  in  depot  maintenance, 
transportation,  and  finance,  gao  found  that  functional  areas  failed  to 
account  for  various  categories  of  significant  costs  when  making  their 
migration  decisions — ^including  costs  related  to  interfacing  migration 
systems  with  other  systems,  project  management,  and  the  system 
selection  process.  More  rigorous  oversight  and  guidance  could  have 
helped  ensure  that  these  costs  were  included. 

GAO  also  found  a  lack  of  good  departmentwide  visibility  over  the  migration 
effort  in  terms  of  project  costs  and  progress.  For  example,  dod  has  not 
been  tracking  key  performance  issues,  such  as  cost  savings  resulting  from 
migration  or  management  and  staff  productivity  improvements.  Further,  it 
does  not  have  a  complete  picture  of  the  costs  of  all  migration  projects  and 
its  scheduling  information  is  inaccurate  and  vmreliable.  As  a  result,  the 
Department  has  not  provided  its  own  department-level  decisionmakers 
with  information  needed  to  manage  the  effort,  and  it  has  not  provided  the 
Congress  with  complete  and  accurate  information  on  migration.  Moreover, 
in  the  absence  of  good  performance  measures  for  migration,  the 
Department  has  not  been  able  to  show  whether  the  overall  strategy  has 
been  successful  or  not. 


Reforming  DOD’s 
Information  Technology 
Investment  Process 
Requires  Effective 
Implementation  of  the 
Chnger-Cohen  Act 


Defense  recognizes  the  need  for  a  better  information  technology 
investment  environment  and  has  taken  steps  to  implement  the 
Clinger-Cohen  Act  of  1996.  The  Congress  passed  the  act  in  an  effort  to  put 
an  end  to  poorly  managed  and  wasteful  information  technology  projects. 
Among  other  things,  it  requires  agencies  to  adopt  an  investment  process 
that  provides  for  the  continual  identification,  selection,  control,  life-cycle 
management,  and  evaluation  of  information  technology  projects.  As  a  first 
step  in  implementing  the  act,  the  Secretary  of  Defense  directed  the  asd 
C3I,  as  the  Department’s  Chief  Information  Officer,  to  take  the  lead  in 
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Recommendations 


implementing  an  investment  process  as  well  as  a  performance-  and 
results-based  management  strategy  for  information  technology. 

This  is  an  important  move  toward  bringing  meaningful  change  to  the 
current  decision-making  and  oversight  environment  for  migration. 
However,  the  current  structure  of  the  cio  position  in  the  Department 
seriously  limits  the  cio’s  ability  to  effectively  serve  as  a  bridge  between  top 
management,  line  management,  and  information  management  support 
officials  and  to  identify  opportunities  to  use  information  technology  to 
enhance  performance.  At  present,  the  asd  C3i  also  serves  as  the 
Department’s  cio.  Asking  the  same  individual  to  serve  in  both  capacities 
prevents  the  cio  from  devoting  full  attention  to  reforming  information 
resources  management  within  the  Department.  It  also  means  that  the  asd 
C3I  wiU  continue  to  be  responsible  for  providing  oversight  for  the  same 
systems  that  he  is  responsible  for  selecting,  developing,  and  implementing 
in  his  capacity  as  a  psa.  This  shortchanges  much-needed  independent 
oversight  for  about  45  percent  of  the  Department’s  migration  system 
investment. 

Finally,  effective  implementation  of  the  Clinger-Cohen  Act  will  not  occur 
unless  Defense’s  information  and  system  investment  control  processes 
successfully  address  the  challenge  posed  by  the  prevailing  organizational 
structure  and  culture  found  throughout  the  Department.  This  condition 
has  promoted  stovepipe  systems  solutions  in  each  component  agency  and 
has  made  it  difficult  to  implement  departmentwide  oversight  or  visibility 
over  information  resources.  This  same  condition  has  contributed  to  the 
difficulty  that  has  limited  the  Department  in  modernizing  business 
processes  and  implementing  corporate  information  systems  across  service 
and  agency  lines.  'This  is  most  evident  in  the  perceived  failure  of  the  ciM 
initiative,  which  was  intended  to  reengineer  business  processes 
throughout  the  Department.  By  doing  so,  the  Department  expected  to  save 
billions  by  having  more  efficient,  effective  business  processes  running 
across  service  and  component  lines.  However,  these  benefits  have  yet  to 
be  widely  achieved  after  8  years  of  effort.  Without  the  Secretary’s  strong 
and  continued  support  for  management  processes  and  controls  designed 
to  improve  information  management  initiatives,  Clinger-Cohen 
implementation  could  well  suffer  similar  results. 


To  ensure  that  continued  investment  in  migration  systems  provides 
measurable  improvements  in  mission-related  and  administrative 
processes,  gao  is  recommending  that  the  Secretary  of  Defense  require 
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Defense  components  to  rank  development/modemization  systems 
justifications  and  complete  them  on  an  expedited  basis,  gao  is  further 
recommending  that  the  Chief  Information  Officer  certify  that  these 
justifications  include  the  following: 

•  An  analysis  of  operational  alternatives  that  clearly  demonstrate  that 
continuing  with  migration  is  the  best  solution  for  improving  performance 
and  reducing  costs  in  the  functional  area. 

•  An  economic  analysis  showing  a  return  on  investment  or  other  mission 
benefits  that  justify  further  investment. 

•  Documentation  showing  that  the  system  currently  complies  with 
applicable  Defense  technical  standards  and  uses  standard  data. 

GAO’s  recommendations  are  also  aimed  at  (1)  correcting  weaknesses 
within  the  current  life-cycle  management  environment  and  (2)  ensuring 
the  successful  reengineering  of  processes  and  implementing  of  corporate 
information  systems  for  functional  areas. 


Agency  Comments 


The  Acting  Assistant  Secretary  of  Defense  for  Command,  Control, 
Communications  and  Intelligence  provided  written  comments  on  a  draft  of 
this  report.  Defense  concurred  with  five  recommendations  and  partially 
concurred  with  two  recommendations.  Defense  did  not  concur  with  the 
remaining  five  recommendations  and  expressed  concerns  about  certain 
aspects  of  the  report.  Generally,  the  Department  concurred  with  gao’s 
recommendations  that  it  revise  or  develop  internal  policies  and 
procedures  to  conform  to  the  Clinger-Cohen  Act,  develop  and  implement 
performance  measures,  and  improve  the  performance  of  internal  tracking 
systems. 

The  Department  did  not  agree  to  a  proposal  in  the  draft  report  that  it  limit 
further  investments  in  ongoing  migration  systems  to  those  that  meet 
critical  needs  imtil  they  are  independently  determined  to  be  economically, 
functionally,  and  technically  justified.  Defense  believed  that  such  a 
limitation  would  not  only  adversely  affect  military  readiness,  system 
development,  and  contract  obligations,  but  also  increase  system 
obsolescence,  gao  understands  dod’s  concerns  regarding  readiness  and 
contract  obligations.  That  is  why  the  recommendation  in  the  draft  report 
recognized  that  critical  needs  still  need  to  be  met.  gao’s  point  is  that 
greater  management  attention  needs  to  be  placed  on  the  migration 
decision-making  process  for  approving  and  funding  the  development  and 
modernization  of  migration  systems  to  achieve  intended  benefits  and 
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minimize  unnecessary  costs.  In  order  to  better  clarify  this  position,  GAO 
reworded  its  recommendation  to  focus  on  strengthening  the  controls  for 
the  decision-making  process  so  that  these  investments  can  be  better 
considered  in  the  Department’s  budget  process. 

Defense  also  did  not  concur  with  a  proposal  in  the  draft  report  that  it 
consider  separating  the  cio  and  the  asd  C3i  positions  so  that  cio  can  devote 
full  attention  to  departmentwide  information  resource  management  issues 
and  provide  independent  oversight.  It  noted,  however,  that  all  asd  C3i 
functions  are  being  reviewed  by  the  Department’s  Task  Force  on  Defense 
Reform  as  part  of  its  review  of  the  Office  of  the  Secretary  of  Defense’s 
organizational  structure,  gao  withdrew  the  proposal  because  dod  is  now 
considering  this  matter.  Nevertheless,  the  concern  that  the  current 
structure  of  the  cio  position  does  not  allow  the  cio  to  devote  full  attention 
to  critical  irm  issues — such  as  computer  security,  the  Year  2000  problem, 
and  the  need  to  develop  and  implement  an  integrated  information 
technology  architectme — ^remains  valid. 

Defense  also  stated  that  the  report  negatively  portrays  Defense’s  efforts  to 
streamline  its  acquisition  processes,  gao  disagrees.  Acquisition 
streamlining  allows  the  Department  to  greatly  simplify  its  acquisition 
processes  for  information  technology  purchases.  However,  acquisition 
streamlining  was  not  intended  to  excuse  dod  from  exercising  management 
controls  to  ensure  that  those  purchases  and  related  system  development 
efforts  made  good  business  sense.  Clinger-Cohen  implementation  should 
allow  the  Department  an  opportunity  to  develop  more  appropriate 
controls. 

Defense’s  comments  are  discussed  in  chapter  4  and  reprinted  in 
appendix  I. 
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Introduction 


Background 


The  Department  of  Defense  (dod)  began  its  Corporate  Information 
Management  initiative  (cim)  in  October  1989  to  help  meet  the  challenge  of 
effectively  managing  its  diverse  operations  as  it  downsized  its  forces  and 
activities.  At  the  time,  the  Department  believed  that  the  thousands  of 
systems  and  numerous  administrative  and  mission-related  processes* 
supporting  dod  functions  were  redundant  and  inefficient  and  that  they 
should  be  standardized  and  made  more  efficient.  To  address  these 
problems,  Defense  planned  to 

simplify  and  improve  business  processes; 

centralize  responsibility  and  authority  in  functional  areas,  such  as  finance, 
personnel,  communications,  health  affairs,  logistics,  command  and 
control,  and  intelligence;  and 

develop  an  integrated  communications  and  data  processing  infrastructure 
based  on  departmentwide  standards. 

From  the  outset,  dod  recognized  that  implementing  cim  would  be  difficult 
because  the  basic  tenet  of  the  initiative — managing  and  implementing 
business  improvements  and  building  corporate  systems  along  functional 
lines — ^represented  a  major  shift  in  the  way  Defense  traditionally  did 
business.  Whereas  each  military  and  Defense  agency  had  historically 
managed  its  own  business  functions  and  information  systems,  cim  called 
on  senior  functional  officials,  known  as  Principal  Staff  Assistants  (psas),^ 
together  with  their  Defense  component  counterparts  to  be  responsible  for 
implementing  improvements  and  information  systems  within  the 
Department’s  business  functions  across  service  and  agency  lines. 

As  CM  was  implemented.  Defense  emphasized  two  ways  of  achieving 
process  improvements  and  addressing  problems  associated  with  its 
disparate  and  stovepiped  information  technology  environment.^  The  first 
was  to  improve,  or  reengineer,  business  processes  first  and  then  apply 
technology  to  the  new  processes.  The  second  involved  selecting  the  best 


‘Also  known  as  business  processes. 

^PSAs  are  the  Under  Secretaries  of  Defense,  the  Director  of  Defense  Research  and  Engineering,  the 
Assistant  Secretaries  of  Defense,  the  Director  of  Operational  Test  and  Evaluation,  the  General  Counsel 
of  DOD,  the  Inspector  General,  the  Assistants  to  the  Secretary  of  Defense  and  the  Office  of  the 
Secretary  of  Defense  (OSD)  Directors  or  equivalents  who  report  directly  to  the  Secretary  or  the 
Deputy  Secretary  of  Defense.  In  essence,  they  are  top  executives  at  the  department  level  who,  together 
with  their  Defense  component  counterparts,  are  charged  with  developing  corporate  systems  and 
reengineering  business  processes  within  their  respective  functional  areas. 

^The  CIM  initiative  included  several  aspects,  such  as  (1)  corporate  policy/planning,  (2)  process  and 
data  modeling,  (3)  process  improvement,  (4)  performance  measurement,  (5)  standard  information 
systems,  and  (6)  computing  and  communications  infrastructure.  DOD  considered  all  the  aspects  of 
CIM  to  be  important  to  effectively  field  information  systems  that  support  its  mission. 
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DOD  information  systems  from  pools  of  existing,  or  legacy,  Systems  that 
provide  similar  automated  support  services  and  eventually  replacing  the 
duplicative  systems  with  the  best  systems.  A  few  years  into  the  ciM  effort, 
this  became  known  as  migration.^  Defense  believed  that  if  done  properly, 
migration  could  cut  costs  associated  with  developing  and  maintaining 
disparate  systems  supporting  the  same  fimctions.  It  also  believed  that 
migration  would  help  to  standardize  business  processes  and  allow  Defense 
to  achieve  savings  that  could  be  put  to  better  use  in  advancing  warfighting 
capabilities.  Figure  1.1  illustrates  how  migration  would  work  in  one 
functional  area. 


'*In  DOD,  migration  can  also  involve  developing  or  acquiring  a  new  system,  rather  than  choosing  from 
existing  systems.  However,  most  migration  efforts  have  involved  choosing  from  existing  systems. 
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Figure  1.1:  An  Example  of  Migration  in 
Practice 


For  its  finance  and  accounting  functions,  Defense's  goal  is  to  reduce  the 
number  of  systems  from  the  324  that  the  services  and  agencies  operated 
in  1991  to  32  systems. 

As  of  the  end  of  fiscal  year  1996,  the  Defense  Finance  and  Accounting 
Service's  strategic  plan  stated  it  had  reduced  the  number  of  finance  and 
accounting  systems  to  217. 

The  migration  strategy  for  accounting  and  finance  is  to 

(1)  identify  a  single  migration  system  for  each  of  the  Department's 
finance  and  accounting  functions,  such  as  civilian  pay, 

(2)  standardize  the  procedures  and  practices  used  to  perform  these 
functions,  and 

(3)  migrate  from  existing  service-unique  systems  to  the  migration 
systems. 

For  some  of  these  functions,  such  as  general  fund  accounting.  Defense 
established  a  two-pronged  approach: 

(1)  migrate  initially  to  a  single  interim  system  or  set  of  systems  within 
each  service  and 

(2)  migrate  from  the  service  systems  to  a  departmentwide  system  or 
set  of  systems. 


Migration  Has 
Become  the  Focal 
Point  for  CIM 


While  Defense  began  cm  emphasizing  the  need  to  improve,  or  reengineer, 
processes  before  appljdng  technology,  it  ended  up  placing  more  priority  on 
obtaining  quick  savings  through  an  accelerated  migration  strategy,  which 
began  in  October  1993.  This  strategy  called  for  selection  of  migration 
systems  in  6  months  and  departmentwide  transition  to  selected  systems  in 
the  following  3  years.  Defense  believed  that  setting  tight  time  frames  for 
migration  with  some  potential  slippage  would  allow  it  to  “harvest  the 
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low-hanging  fruit”  of  potential  savings  before  reengineering.  By  default, 
this  meant  that  the  increased  emphasis  on  migration  would  postpone  the 
dramatic  gains  that  could  be  achieved  through  reengineering.® 

The  risks  involved  with  postponing  reengineering  efforts  were  significant. 
Reengineering  identifies,  analyzes,  and  redesigns  an  organization’s  core 
business  processes,  aiming  to  achieve  dramatic  improvements  in  critical 
areas  of  performance,  such  as  cost,  quality,  service,  and  speed.  It  focuses 
on  redesigning  the  business  process  as  a  whole  in  order  to  achieve  the 
greatest  possible  benefits  to  an  organization  and  its  customers.  Migration, 
on  the  other  hand,  focuses  on  standardizing  existing  processes  and 
information  systems.  This  can  yield  some  benefits,  such  as  reducing  the 
need  to  maintain  disparate  systems  that  support  the  same  functions,  but 
seldom  yields  dramatic  improvements.  In  addition,  if  the  process  is 
inefficient  or  outmoded,  migration  only  serves  to  perpetuate  a  bad 
process.  Finally,  choosing  migration  before  reengineering  may  cause 
future  reengineering  efforts  to  be  more  difficult  by  entrenching  inefficient 
and  ineffective  work  processes. 

We  raised  these  concerns  both  before®  and  after^  Defense  decided  to 
embark  on  the  accelerated  migration  strategy.  We  believed  that  the  shift  in 
emphasis  toward  migration  seriously  endangered  cm’s  chances  for 
success.  For  this  reason,  and  because  ciM-related  technology  investments, 
which  are  expected  to  total  billions  of  dollars  each  year,  are  vulnerable  to 
waste  and  mismanagement,  we  designated  cm  as  a  high-risk  government 
information  technology  initiative.®  Nevertheless,  Defense  proceeded  to 
concentrate  on  migration  even  in  areas  where  it  had  recognized  that  there 
was  a  significant  need  to  change  business  processes.  One  such  area  was 
transportation.  We  reported  in  1996  that  even  though  Defense  recognized 
that  its  military  transportation  processes  were  fragmented,  outdated. 


^In  an  October  1993  memorandum,  the  Deputy  Secretary  of  Defense  announced  a  near-term  strategy 
that  focused  on  migration  and  data  standardization.  The  memorandum  stated  that  completion  of  other 
initiatives  were  not  to  be  prerequisites  of  implementation  of  migration  systems  and  data  standards. 
While  some  functional  areas  may  have  continued  process  reengineering  initiatives,  our  work  indicates 
that  many  others  gave  priority  to  the  migration  strategy. 

‘^Defense  ADP:  Corporate  Information  Management  Must  Overcome  Major  Problems 
(GAO/IMTEC-92-77,  September  14, 1992). 

^Defense  Management:  Stronger  Support  Needed  for  Corporate  Information  Management  Initiative  to 
Succeed  rGAO/AIMD/NSIAD-94-101,  April  12, 1994)  and  Defense  Management:  Impediments 
Jeopardize  Logistics  Corporate  Information  Management  (GAO/NSIAD-95-28,  October  21, 1994). 

®Our  high-risk  effort,  which  began  in  1990,  identifies  those  federal  program  areas  we  consider  high  risk 
because  they  are  especially  vulnerable  to  waste,  fraud,  abuse,  and  mismanagement.  See  High-Risk 
Series:  Information  Management  and  Technology  (GAO/HR-97-9,  February  1997)  and  High-Risk  Series: 
An  Overview  (GAO/HR-95-1,  February  1995)  for  a  discussion  on  CIM. 
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Status  of  the  Migration 
Effort 


inefficient,  and  costly,  it  focused  on  technology  solutions  rather  than  the 
need  to  identify  and  correct  the  root  causes  of  its  transportation  problems.^ 

Because  reengineering  took  a  backseat  to  migration  in  other  business 
areas  as  well,  cim  has  never  been  able  to  achieve  the  level  of  cost  savings 
and  process  improvements  originally  intended.  A  recent  report^® 
conducted  by  RAND,  a  consulting  organization,  notes  that  “The  cim  effort 
is  today  widely  viewed  as  a  failure  in  most  quarters  of  the  dod.  It  has  not 
resulted  in  either  significant  process  reengineering  or  visible  savings  in  the 
hardware  and  software  required  to  support  all  the  varied  information 
systems  in  the  Defense  infrastructure.” 

Defense  is  still  planning  to  invest  at  least  $18  billion  in  its  migration  effort 
from  fiscal  years  1995  through  2000.  Because  this  investment  is  significant, 
we  were  asked  for  information  on  the  status  and  progress  of  migration  and 
whether  Defense  has  significant  controls  in  place  to  manage  and  oversee 
the  effort. 


Defense  reported  that  it  has  selected  363  systems  for  migration.  Forty-nine 
of  the  363“  migration  systems  are  large-scale,  or  major,  systems^^  that  are 
expected  to  cost  at  least  $13  billion — or  about  72  percent  of  the  total 
$18  billion  cost  estimate — ^to  develop,  deploy,  and  maintain  over  fiscal 
years  1995  through  2000.  Defense  has  identified  1,938  legacy  systems  for 
potential  termination.  As  of  April  1996,  its  Defense  Integration  Support 
Tools  (dist)  database  showed  that  281  had  already  been  terminated,  886 
were  scheduled  for  termination  by  the  year  2000,  and  771  more  were 
identified  for  potential  elimination  after  the  year  2000. 

Table  1.1  provides  the  number  of  systems  selected  by  each  functional  area, 
the  number  of  legacy  systems  identified  for  potential  termination,  and 
dod’s  estimated  costs  of  developing,  deploying,  and  maintaining  the 
migration  systems.  We  excluded  costs  for  27  systems  for  which  dod 


^Defense  Transportation:  Migration  Systems  Selected  Without  Adequate  Analysis  (GAO/AIMD-96-81, 
August  29, 1996). 

^°Strategic  Appraisal  1997:  Strategy  and  Defense  Planning  for  the  21st  Century,  February  1997,  RAND’s 
Project  Air  Force;  edited  by  Zalmay  M.Khalilzad  and  David  A  Ochmanek. 

^^The  numbers  of  migration  systems  selected  and  legacy  systems  identified  for  potential  termination 
are  subject  to  change  as  PSAs  make  additional  selections  and  reevaluate  previous  ones.  Also,  the  PSAs 
classified  138  of  the  363  migration  systems  as  interim  systems. 

^-In  DOD,  major  information  systems  projects  are  those  that  (1)  have  estimated  program  costs  in 
excess  of  $30  million  in  any  1  year,  (2)  have  estimated  program  costs  of  over  $120  million  in  total, 

(3)  have  total  life-cycle  costs  of  over  $360  million,  or  (4)  are  designated  as  being  of  special  interest. 
DOD  periodically  revises  these  dollar  thresholds. 
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collected  costs  because  the  systems  were  classified.  And  costs  for  the 
remaining  121  of  363  systems  were  not  included  in  the  table  because  the 
Office  of  the  asd  C3i  had  not  collected  the  costs  for  these  systems.  Table 
1.2  provides  the  name,  status,  and  cost  for  each  of  the  49  major  systems. 


Table  1.1 :  Status  Reported  to  GAO  on  Defense  Migration  Systems  Seiections  and  Costs 

Dollars  in  millions 

Functional  area 

Number  of  legacy 
systems  identified 
for  potential 
termination 

Number  of  migration/ 
interim  systems 
selected 

Number  of  migration/ 
interim  systems 
reporting  costs 

Total  cost  reported  to 
GAO  for  fiscal  years 
1995  through  2000 

Command  and  Control 

67 

35 

34 

4.493.2 

intelligence 

304 

69 

46 

572.6" 

Mission  Support 

Atomic  Energy 

7 

8 

2 

12.7 

Communications 

17 

2 

2 

3,483.2 

Environmental  Security 

286 

11 

6 

33.1 

Finance‘s 

267 

52 

31 

1,417.1 

Health 

62 

55 

49 

2.533.8 

Human  Resources 

198 

20 

10 

933.1 

Information  Management 

21 

5 

0 

0.0 

Inspector  General 

8 

2 

1 

15.8 

Logistics 

496 

63 

53 

4.459.1 

Meteorology  and 
Oceanography 

33 

33 

0 

0.0 

Policy 

3 

7 

7 

73.9 

Procurement 

67 

1 

1 

282.1 

Science  and  Technology 

102^ 

0" 

0" 

c 

Total 

1,938 

363 

242 

$18,309.7 

Note:  We  did  not  independently  verify  information  provided  in  this  table. 


^Costs  were  not  included  for  27  systems  for  which  DOD  collected  cost  data  because  the  systems 
were  classified. 

‘^The  information  presented  for  finance  is  not  consistent  with  information  presented  in  figure  1.1  of 
this  chapter  because  the  counts  of  systems  occurred  at  different  points  in  time. 

'^The  science  and  technology  functional  area  has  not  yet  selected  migration  systems  and  has  not 
specified  which,  if  any,  of  its  legacy  systems  may  be  terminated. 

Source:  Information  obtained  from  the  DIST,  DOD’s  April  1996  Report  to  the  Congress, 
discussions  with  Defense  functional  area  managers,  and  the  asd  C3i.  Cost  data  were  provided  as 
of  December  1996. 
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Table  1.2:  Major  Migration  and  Interim  Systems/Applications  Selected  for  Each  Functional  Area 

Dollars  in  millions 

Total  cost  data 


System  name 

Acronym 

Current 

milestone^ 

reported  to  GAO  for 
fiscal  years  1995 
through  2000 

Systems  being  reviewed  by  the  Defense  Acquisition  Board  (six  systems): 

Command  and  Control  systems 

1  Advanced  Field  Artillery  Tactical  Data 

System 

AFATDS 

3 

$  320.6 

2  Combat  Service  Support  Control  System 

csscs 

2 

104.9 

3  Forward  Area  Air  Defense  C2 

Intelligence  System 

FAADC2I 

3 

216.4 

4  Maneuver  Control  System 

MCS 

3 

253.7 

Intelligence  systems 

5  All  Source  Analysis  System 

ASAS 

3 

347.9 

6  Joint  Service  Imagery  Processing  System  JSIPS 

2 

b 

Systems  being  reviewed  by  the  Major  Automated  Information  System  Review  Council  (43  systems): 

Command  and  Control  systems 

1  Air  Force  Mission  Support  System 

AFMSS 

3 

351.9 

2  Army  Global  Command  and  Control 

System 

AGCCS 

3 

254.6 

3  Counter  Narcotics  Command 

Management  Control  System 

CN/CMS 

3 

112.2 

4  Global  Command  and  Control  System 

GCCS 

3 

422.1 

5  Naval  Aviation  Logistics  Command 

Management  Information  System 

NALCOMIS 

3 

93.5 

6  Naval  Tactical  Command  System  -  Afloat 

NTCS-A 

3 

314.1 

7  Operations  Support  System 

OSS 

3 

174.1 

8  Shipboard  Non-Tactical  ADP  Program  III 

SNAP  III 

3 

401.4 

9  Strategic  War  Planning  System 

SWPS 

3 

343.4 

10  Tactical  Support  Center 

TSC 

3 

154.3 

Intelligence  systems 

1 1  Exploitation  Support  System 

ESS 

0 

b 

12  High  Performance  Computing 

Modernization  Program 

HPCMP 

2 

b 

Mission  support  systems 

Communications 

13  Defense  Information  System  Network 

DISN 

0 

2,556.2 

14  Defense  Message  System 

DMS 

2 

927.0 

(continued) 
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Dollars  in  millions 

System  name 

Acronym 

Current 

milestone* 

Total  cost  data 
reported  to  GAO  for 
fiscal  years  1995 
through  2000 

Finance 

15 

Defense  Joint  Military  Pay  System 

DJMS 

1/2 

114.9 

16 

Defense  Procurement  Payment  System 

DPPS 

1/2 

68.5 

17 

Standard  Accounting  And  Reporting 
System 

STARS 

2/3 

263.7 

Health 

18 

Ambulatory  Data  System 

ADS 

3 

90.3 

19 

Composite  Health  Care  System 

CHCS 

3 

785.6 

20 

Composite  Health  Care  System  11 

CHCS  II 

0 

1.4 

21 

Corporate  Executive  Information  System 

CEIS 

0/1/2 

18.2 

Human  resources 

22 

Defense  Civilian  Personnel  Data  System 

DCPDS 

1 

58.5 

23 

Defense  Commissary  Information  System  DCIS 

2 

120.8 

24 

Defense  Commissary  Point  of  Sales 
System 

PCS 

3 

262.4 

25 

Joint  Recruiting  Information  Support 
System 

JRISS 

0 

207.0 

26 

Navy  Standard  Integrated  Personnel 
System 

NSIPS 

0 

b 

27 

Reserve  Component  Automation  System 

RCAS 

2/3 

b 

28 

standard  Installation/Division  Personnel 
System  -  3 

SIDPERS-3 

2 

b 

Loqlstlcs 

29 

Ammunition  Management  Standard 
System 

AMSS 

0 

55.7 

30 

Command  and  Control  Information 
Processing  System 

C2IPS 

3 

311.8 

31 

Defense  Automatic  Addressing  System 

DAAS 

3 

12.7 

32 

Defense  Medical  Logistics  Standard 
Support  System 

DMLSS 

2 

129.0 

Department  of  Army  Movements 
Management  System  -  Redesigned 

DAMMS-R 

3 

43.9 

34" 

Depot  Maintenance  System 

DMS 

0" 

438.6 

35 

Distribution  Standard  System 

DSS 

3 

270.2 

36 

Global  Transportation  Network 

GTN 

2 

179.5 

Joint  Computer-Aided  Acquisition  and 
Logistics  System 

JCALS 

2 

789.0 

Joint  Engineering  Data  Management 
Information  and  Control  System 

JEDMICS 

3 

243.7 

/  1.: _ 

(continued) 
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Dollars  in  millions 

System  name 

Acronym 

Current 

milestone^ 

Total  cost  data 
reported  to  GAO  for 
fiscal  years  1995 
through  2000 

39 

Materiel  Management  System 

MMS 

0" 

1,149.2 

40 

Transportation  Coordinators’  Automated 
Information  for  Movement  System  II 

TC-AIMS  II 

0 

15.1 

41 

Transportation  Operational  Personal 
Property  System 

TOPS 

3 

95.4 

Meteorology  &  Oceanography 

42 

Primary  Oceanographic  Prediction 
System 

POPS 

3 

b 

Procurement 

43 

DOD  Standard  Procurement  System 

SPS 

1 

282.1 

Total 

$13,355.5 

Note:  We  did  not  verify  the  information  provided  in  this  table.  As  discussed  in  chapter  2,  these 
costs  may  be  inaccurate. 

®There  are  four  milestones  in  the  major  system  acquisition  review  process.  These  are  milestone 
0 — approval  to  conduct  concept  studies;  milestone  1 — approval  to  begin  a  new  acquisition 
program;  milestone  2 — approval  to  enter  engineering  and  manufacturing  development;  and 
milestone  3~approval  to  produce,  field,  or  deploy  the  system.  Multiple  milestones  may  be  shown 
for  some  projects  because  the  projects’  subsystems  or  applications  are  in  different  phases  of  the 
acquisition  process. 

^The  Office  of  the  ASD  C3I  did  not  collect  costs  for  this  system. 

'"Both  of  these  programs  were  redirected  and  are  no  longer  targeted  to  be  standard  migration 
systems. 

Source:  Information  DOD  reported  to  the  Congress  in  April  1996  and  to  GAO  in  the  course  of  this 
audit. 


Objectives,  Scope, 
and  Methodology 


We  were  asked  (1)  for  information  on  the  number  and  cost  of  systems 
designated  for  migration,  the  number  of  legacy  systems  already  terminated 
or  scheduled  for  termination,  and  savings  resulting  from  terminations  of 
legacy  systems  and  (2)  whether  Defense’s  management  control  and 
oversight  processes  for  migration  systems  are  ensuring  that  the 
investments  are  economically  sound  and  in  compliance  with  Defense’s 
technical  and  data  standards. 

To  assess  the  status  of  Defense’s  overall  migration  strategy  and  obtain 
available  information  on  the  number  and  cost  of  systems  designated  for 
migration,  the  number  of  legacy  systems  cilready  terminated  and  scheduled 
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for  future  termination,  and  savings  resulting  from  terminations  of  legacy 
systems,  we  analyzed  a  Defense  April  1996  report  to  the  Congress 
containing  information  on  the  schedule,  cost,  and  status  of  the  migration 
systems.  Defense  prepared  this  report^^  in  response  to  a  requirement  in 
Section  366  of  the  National  Defense  Authorization  Act  for  fiscal  year  1996. 
We  also  obtained  and  analyzed  a  copy  of  the  dist  database  as  of  April  1996 
that  Defense  used  to  develop  the  report  to  the  Congress,  as  well  as  to 
provide  its  own  senior  managers  information  on  migration  system 
selections  and  legacy  system  terminations.  Additionally,  we  reviewed — ^but 
did  not  independently  verify — cost  information  that  Defense  reported  to 
Congress  as  part  of  the  April  1996  Section  366  report.  We  also  interviewed 
senior  dod  officials  to  determine  if  additional  cost  and  performance 
information  on  migration  systems  existed  at  the  Office  of  the  Secretary  of 
Defense  (osd)  level. 

We  assessed  the  reliability  of  the  data  presented  to  the  Congress  in  two 
ways.  First,  we  identified  missing  and  conflicting  information  in  the 
April  1996  Section  366  report  to  the  Congress  and  requested  that  the  asd 
C3I  staff  to  clarify  conflicting  information.  Second,  we  compared  the  data 
reported  to  the  Congress  for  three  functional  activities  (business 
areas) — ^Transportation,  Civilian  Personnel,  and  Clinical  Health — against 
data  provided  to  us  by  the  functional  area  managers  at  these  activities.  We 
then  established  and  analyzed  a  database  containing  schedule  and 
available  cost  information  for  the  migration  and  legacy  systems  and 
modified  the  database  to  reflect  updated  schedule,  costs,  and  other 
descriptive  data  Defense  provided  to  us  during  the  course  of  the  audit. 

To  assess  Defense’s  management  control  and  oversight  processes  for 
migration  systems  to  determine  whether  the  investments  are  economically 
justified  and  comply  with  Defense  technical  and  data  standards,  we 
reviewed  the  department-level  management  and  oversight  processes  for 
approving  the  psas’  migration  system  selections  and  for  overseeing  major 
migration  systems’  acquisitions  by  the  Major  Automated  Information 
System  Review  Council  (maisrc).‘^  Our  review  did  not  focus  on  the  process 
by  which  the  psas  select,  develop,  and  manage  their  migration  systems. 


noted  in  this  chapter,  this  report  did  not  include  costs  for  121  information  systems. 

^^MAISRC  provides  acquisition  oversight  for  information  systems  that  (1)  are  anticipated  to  cost 
$30  million  or  more  a  year,  (2)  have  estimated  program  costs  in  excess  of  $120  million,  (3)  have 
estimated  life-cycle  costs  of  more  than  $360  million,  or  (4)  are  designated  for  review  by  the  ASD  C3I, 
who  also  chairs  the  MAISRC.  MAISRC  reviews  such  matters  as  whether  a  proposed  system  is  being 
developed  in  accordance  with  Defense  policies,  procedures,  and  regulations  and  whether  system 
managers  took  steps  to  minimize  the  cost  of  a  new  system. 
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Nor  did  we  examine  acquisition  review  processes  within  the  individual 
functional  areas  employed  for  nonmagor  system  projects. 

To  analyze  Defense’s  oversight  processes  for  reviewing  and  approving  the 
PSAs’  migration  system  selections,  we  reviewed  department-level  approval 
of  the  PSAs’  selections.  We  obtained  a  list  of  migration  systems  that  psas 
had  selected  for  their  functional  areas.  We  then  visited  the  Office  of  the 
ASD  C3I  and  the  Defense  Information  Systems  Agency  (disa)  and  reviewed 
the  business  case  analysis  documentation,  technical  analysis 
documentation,  and  other  documentation  provided  to  those  offices  for 
department-level  approval  of  the  psa’s  migration  and  interim  system 
selections.  We  also  interviewed  officials  from  the  Office  of  the  asd  C3I, 

DISA,  and  the  offices  of  selected  psas  regarding  the  documentation 
supporting  the  selections. 

To  review  Defense’s  acquisition  oversight  processes  for  major  migration 
systems,  we  first  identified  the  major  migration  systems,  as  defined  by 
Defense  regulations.  We  then  obtained  and  analyzed  the  progress  reports 
and  other  documentation  provided  to  maisrc  for  those  systems.  We  also 
interviewed  Defense  representatives  and  officials  responsible  for 
information  systems  oversight  in  the  Office  of  the  asd  C3i  and  offices  of 
DISA,  the  Defense  Acquisition  Board  (dab),  maisrc,  and  Program  Analysis 
and  Evaluation  (pa&e).  We  also  interviewed  representatives  of  the  program 
offices  or  oversight  offices  for  selected  systems  for  which  oversight  had 
been  delegated  by  maisrc.  Through  document  reviews  and  interviews,  we 
determined  whether  economic  analyses  for  major  migration  systems  had 
been  independently  reviewed  and  if  so,  whether  the  reviews  had  identified 
problems  in  the  analyses.  We  also  determined  whether  dod  guidance 
existed  on  preparing  economic  analyses  for  information  systems.  A  gao 
economist  met  with  pa&e  analysts,  reviewed  problems  that  were  identified 
with  economic  analyses,  and  verified  the  validity  of  the  problems  reported. 

Additionally,  we  determined  whether  maisrc  had  information  on 
alternative  analyses  performed  for  major  migration  system  selections. 
Lastly,  we  determined  whether  maisrc  has  sufficient  oversight  information 
to  ensure  that  major  migration  systems  are  complying  with  applicable 
technical  and  data  standards  that  are  necessary  to  achieve  an 
interoperable  systems  environment. 

We  reviewed  Defense’s  policies  and  guidance  for  migration  systems  to 
ensure  that  information  technology  is  acquired,  managed,  and  used  in  the 
most  efficient  and  effective  manner.  Our  assessment  included  analyzing 
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the  National  Defense  Authorization  Acts,  Committee  reports,  and 
Conference  reports  for  fiscal  years  1993  through  1997;  Defense  Office  of 
Inspector  General  reports;  our  prior  studies  of  cim  and  the  migration 
system  strategy,  other  available  evaluations  of  the  cim  and  the  migration 
strategy,  and  existing  legislation  affecting  these  efforts.  The  major  studies 
reviewed  included  studies  by  RAND  and  the  Defense  Science  Board. 
Pertinent  existing  legislation  includes  the  Clinger-Cohen  Act,  the 
Government  Performance  and  Results  Act,  the  Paperwork  Reduction  Act, 
and  the  Chief  Financial  Officers  Act. 

We  conducted  our  review  from  August  1996  through  July  1997  in 
accordance  with  generally  accepted  government  auditing  standards.  We 
requested  comments  on  a  draft  of  this  report  from  the  Department  of 
Defense.  The  Acting  Secretary  for  Command,  Control,  Communications 
and  Intelligence  provided  us  with  written  comments.  These  comments  are 
discussed  in  chapter  4  and  reprinted  in  appendix  I. 


‘^Achieving  an  Innovative  Support  Structure  for  21st  Century  Military  Superiority:  Higher  Performance 
at  Lower  Costs,  1996  Summer  Study,  Defense  Science  Board,  Department  of  Defense.  The  Defense 
Science  Board  is  a  Federal  Advisory  Committee  established  to  provide  independent  advice  to  the 
Secretary  of  Defense.  Statements,  opinions,  conclusions,  and  recommendations  in  their  reports  do  not 
necessarily  represent  the  official  position  of  DOD. 
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Management  Control  Processes  Over 
Individual  Migration  Efforts  Have  Broken 
Down 


Embarking  on  the  migration  strategy  was  an  extremely  risky  endeavor  for 
the  Department  of  Defense.  First,  in  developing  standard  systems  within 
functional  areas,  the  Department  had  to  carefully  consider  complex 
technical  issues,  such  as  interfacing  migration  systems  with  other  systems 
and  contending  with  nonstandard  data  formats  and  definitions.  For 
Defense,  such  complexities  were  compoimded  by  its  sheer  size  and  the 
numbers  of  disparate  systems.  We  believe  an  even  tougher  obstacle  facing 
Defense,  however,  was  the  prevailing  culture,  which  was  based  on 
decentralized  department  organizational  structure,  nonstandard  processes 
and  procedures.  In  general,  each  military  service  and  Defense  agency  has 
historically  managed  its  own  business  functions  and  information 
technology  projects,  whereas  cim  and  migration  required  business 
improvements  and  information  systems  to  be  managed  on  a 
Departmentwide,  or  corporate  basis. 

Therefore,  to  ensure  the  strategy’s  success,  it  was  vital  for  Defense  to 
establish  an  effective  decision-making  and  oversight  environment  for 
making  migration  investments.  It  would  need  controls  and  processes  that 
ensured  inefficient  administrative  and  mission-related  work  processes 
were  modernized  before  significant  technology  investments  were  made  to 
support  them.  It  would  also  need  controls  that  ensured  that  the  migration 
projects  themselves  were  effectively  managed  as  investments  so  that  the 
Department  could  target  resources  and  attention  to  priority  areas  and  stop 
those  projects  that  failed  to  meet  their  goals.  Finally,  Defense  needed  life 
cycle  management  controls  that  ensured,  on  a  system-by-system  basis, 
that  sound  management  and  development  practices  were  followed. 

However,  Defense’s  primary  corporate-level  control  mechanism  for 
ensuring  that  sound  management  and  development  practices  were 
followed  for  each  migration  investment  has  not  been  effective.  This 
control  requires  that  all  selections  to  be  submitted  for  approval  by  the 
Assistant  Secretary  of  Defense  for  Command,  Control,  Communications 
and  Intelligence  (asd  C3I),  who  is  also  the  Department’s  Chief  Information 
Officer  (cio).  In  approving  the  systems,  the  Assistant  Secretary  is  to  review 
data,  technical,  and  programmatic  factors  relating  to  the  selection.  We 
found,  however,  that  most  migration  selections  never  came  in  for  this 
approval  and  those  that  did  were  approved  in  the  absence  of  critical 
technical  and  programmatic  supporting  documents.  (A  description  of  the 
migration  decision  process  is  provided  in  appendix  n.) 

Because  this  control  was  playing  a  marginal  role  in  ensuring  the  success  of 
migration,  we  assessed  whether  Defense’s  acquisition  oversight 
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processes — which  are  designed  to  augment  management  oversight  for  the 
Department’s  most  expensive  information  systems — ^were  helping  to 
ensure  that  the  major  migration  investments  were  economically  and 
technically  sound.  These  processes  have  also  broken  down  where 
migration  is  concerned.  Had  there  been  more  rigorous  attention  to 
oversight  in  both  areas,  Defense  may  well  have  avoided  the  migration 
problems  we  identified  in  previous  reviews  that  cost  the  Department 
hundreds  of  millions  of  dollars. 

In  implementing  the  migration  strategy,  Defense  did  not  ensure  that  it  had 
adequate  visibility  over  status,  costs,  and  progress.  As  a  result,  it  has  not 
been  able  to  (1)  demonstrate  whether  the  migration  strategy  has  been 
successful,  (2)  provide  the  Congress  with  accurate  and  reliable 
information  needed  for  oversight  purposes,  and  (3)  provide  its  own 
decisionmakers  at  the  headquarters  level  with  information  needed  to 
oversee  the  strategy. 


Most  Systems  Not 
Approved  by 
Defense’s  Senior 
Technology  Manager 


After  selecting  a  migration  system,  the  Principal  Staff  Assistant  (psa)* 
responsible  for  a  functional  area  is  to  submit  the  system  for  approval  by 
the  Assistant  Secretary  of  Defense  for  Command,  Control, 
Communications  and  Intelligence — ^the  Senior  Information  Management 
Official  and  Chief  Information  Officer  in  dod.  In  his  memorandum^  setting 
forth  the  requirements  for  selecting  and  reviewing  migration  systems,  the 
ASD  C3I  stated  that  in  approving  the  selections,  he  would  consider  data, 
technical,  and  programmatic  factors.  For  example,  the  asd  C3i  review 
would  include  such  issues  as  whether  the  migration  systems  will  lend 
themselves  to  data  sharing  and  whether  they  conform  to  dod’s  technical 
standards— which  act  as  a  set  of  “building  codes”  for  constructing  systems 
to  ensure  they  will  be  compatible  with  its  information  infrastructure  and 


^The  PSA  is  the  senior  executive-level  manager  who  is  responsible  for  the  management  of  a  defined 
function  or  functions  within  DOD. 

-Requirements  for  selecting  and  reviewing  migration  systems  under  DOD’s  accelerated  migration 
strategy  are  described  in  two  memorandums.  The  first  memorandum  was  issued  by  the  Deputy 
Secretary  of  Defense  in  October  1993,  and  the  second  was  issued  by  the  Assistant  Secretary  of  Defense 
for  Command,  Control,  Communications  and  Intelligence  in  November  1993. 
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technically  interoperable  with  each  other.^  This  approval  is  important 
because,  as  the  Department’s  Chief  Information  Officer,  the  Assistant 
Secretary  needs  to  ensure  that  the  migration  systems  help  facilitate  the 
sharing  of  information  across  service  and  agency  lines. 

However,  both  the  psas  and  the  asd  C3i  did  not  adhere  to  this  oversight 
process.  First,  245  selections,  or  about  67  percent,  were  never  submitted 
for  this  oversight  even  though  all  363  system  selections  should  have 
been.^'  ®  Of  the  49  major,  or  large  scale,  migration  systems,  only  16  were 
submitted  for  this  review.  Thus,  for  the  bulk  of  migration  systems,  the 
Assistant  Secretary  was  unable  to  ensure  that  Defense  technical  standards 
would  be  met  and  that  the  best  system  development  practices  were  being 
followed. 

Second,  most  of  the  systems  that  were  submitted  for  this  review  were 
approved  in  the  absence  of  critical  technical  and  programmatic  business 
case  support.  For  example,  23  of  the  118  selections  that  were  submitted 
for  this  oversight,  about  19  percent,  were  approved  without  documents 
stating  that  the  functional  area  complied  or  planned  to  comply  with 
technical  standards.  In  addition,  64  of  the  118  selections,  about  54  percent, 
were  approved  without  documents  showing  that  the  psa  responsible  for 
the  functional  area  performed  some  type  of  functional  economic  analysis 
or  other  similar  business  case  analysis.  This  analysis  is  the  means  by 
which  Defense  managers  are  supposed  to  evaluate  different  options  for 
improving  business  areas,  such  as  outsourcing,  reengineering,  or 
migration.  Because  all  of  the  migration  selections  submitted  for  this 


^Defense’s  technical  and  data  standards  are  designed  to  enable  systems  to  easily  interoperate  and 
transfer  information.  Its  standard  definitions  for  data  elements  are  intended  to  ensure  that  users  of  all 
Defense  systems  define  the  same  data  in  the  same  way  and  have  a  common  understanding  of  their 
meaning.  Defense  has  developed  or  is  in  the  process  of  defining  technical  standards  in  the  Technical 
Architecture  FYamework  for  Information  Management  (TAFIM),  the  Joint  Technical  Architecture 
(JTA),  and  the  Defense  Information  Infrastructure  Conunon  Operating  Environment  (dii  coe).  The 
Defense  Information  Systems  Agency  (DISA)  is  responsible  for  developing,  obtaining  from  commercial 
sources,  and  maintaining  the  compilation  of  Defense  Information  Infrastructure  technical  standards, 
and  it  is  responsible  for  maintaining  a  Defense  data  dictionary  system  as  a  repository  of  data 
requirements  and  for  facilitating  the  cross-functional  coordination  and  approval  of  standard  formats, 
definitions,  etc.  PSAs,  the  military  services,  Defense  agencies,  and  Joint  Chiefs  of  Staff  are  responsible 
for  reaching  agreement  on  the  standards  and  approving  them  as  DOD  standard  data  elements.  DISA  is 
then  responsible  for  disseminating  the  approved  standard  data  elements  for  use  throughout  the 
Department. 

“^Nine  of  the  systems  that  were  submitted  for  approval  were  grandfathered  into  approval  because 
development  for  them  was  well  underway  at  the  time  the  accelerated  migration  system  strategy  began 
in  1993. 

•^Forty-two  of  the  systems  that  were  approved  by  the  ASD  C3I  as  migration  systems  did  not  follow 
DOD’s  migration  system  approval  process  described  in  appendix  II,  Instead,  the  ASD  C3I  approved 
these  42  Command  and  Control  systems  based  on  reviews  by  the  Military  Communications  Electronics 
Board. 


Page  30 


GAO/AIMD-98-5  Defense  Migration  Strategy 


Chapter  2 

Management  Control  Processes  Over 
Individual  Migration  Efforts  Have  Broken 
Down 


oversight  have  been  approved — ^whether  adequately  supported  by 
technical  and  business  case  justification  or  not — the  asd  C3i  oversight  has 
essentially  become  a  meaningless  process. 


Acquisition  Oversight 
Process  Did  Not 
Ensure  System 
Selections  Were 
Economically  and 
Technically  Sound 


Because  the  asd  C3i  oversight  control  over  the  migration  strategy  played  a 
marginal  role  in  ensuring  that  sound  management  and  development 
practices  are  followed,  we  assessed  whether  Defense’s  traditional  major 
information  systems  acquisition  oversight  processes  were  doing  so  for  43 
of  the  major  migration  systems.®  These  review  processes  are  designed  to 
assess  whether  projects  8ire  affordable  and  financial  and  operational  risks 
have  been  minimized. 

Once  under  acquisition  oversight,  systems  are  normally  reviewed  and 
approved  at  each  of  four  milestones.'^  The  reviews  involve  assessing  such 
matters  as  whether  the  proposed  system  is  being  developed  in  accordance 
with  Defense  policies,  procedures,  and  regulations;  whether  the  systems’ 
program  managers  took  steps  to  minimize  the  cost  of  a  new  system  by 
ensuring  full  and  open  competition;  and  whether  the  program  managers 
will  effectively  use  advanced  system  design  and  software  engineering 
technology  to  minimize  software  and  maintenance  costs. 


However,  these  reviews  have  not  been  effective  for  the  migration  effort. 
As  discussed  in  the  following  section,  maisrc  could  not  provide  sufficient 
assurance  that  the  major  migration  systems  are  economically  justified  and 
comply  with  Defense’s  technical  and  data  standards. 


Ineffective  Review  and 
Preparation  of  Economic 
Analyses 


A  principal  tool  of  acquisition  oversight  is  the  economic  analysis  because 
it  helps  to  ensure  that  the  system  chosen  for  development  is  cost-effective. 
If  done  properly,  it  can  enable  reviewers  to  determine  whether  dod  has 


®As  noted  earlier,  there  are  49  major  migration  systems.  Six  of  these  are  under  the  oversight  of  the 
Defense  Acquisition  Board  (DAB).  This  is  Defense’s  senior-level  forum  for  advising  the  Under 
Secretary  of  Defense  (Acquisition  and  Technology)  on  critical  decisions  concerning  major  weapon 
systems  and  large-scale  information  systems,  principally  in  the  communication,  command,  control, 
and  intelligence  functional  areas.  Criteria  for  DAB  oversight  are  (1)  estimated  expenditures  for 
research,  development,  test,  and  evaluation  of  more  than  $355  million  or  (2)  estimated  procurement 
costs  of  more  than  $2,135  billion.  One  difference  between  the  DAB  and  MAISRC  oversight  processes  is 
that  systems  reviewed  by  DAB  are  required  to  have  independent  cost  estimates  rather  than  economic 
analyses.  According  to  the  DAB  analyst  responsible  for  coordinating  oversight  of  these  systems, 
Program  Analysis  and  Evaluation  (PA&E)  analysts  performed  independent  cost  estimates  for  all  six 
migration  systems  under  DAB  oversight. 

^The  four  milestones  are:  milestone  0,  approval  to  conduct  concept  studies;  milestone  1,  approval  to 
begin  a  new  acquisition  program;  milestone  2,  approval  to  enter  engineering  and  manufacturing 
development;  and  milestone  3,  approval  to  produce,  field,  or  deploy  the  system. 
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sufficient  funds  in  its  budget  to  pay  for  the  system,  that  is,  whether  the 
system  is  affordable.  It  can  also  reveal  potential  conflicts  between 
available  funding  and  the  planned  schedule  for  deployments.  To  arrive  at 
these  conclusions,  the  economic  analysis  establishes  baseline  life-cycle 
costs  and  benefit  estimates  for  the  project  and  calculates  the  project’s 
return  on  investment. 

The  importance  of  developing  complete  and  accurate  economic  analyses 
is  underscored  by  several  govemmentwide  requirements.  For  example,  the 
Office  of  Management  and  Budget’s  (omb)  Circular  A-130,  Management  of 
Federal  Information  Resources,  calls  on  agencies  “to  conduct  benefit-cost 
analyses  to  support  ongoing  management  oversight  processes  that 
maximize  return  on  investment  and  minimize  financial  and  operational 
risks  for  investments  in  major  information  systems  on  an  agencywide 
basis.”  Likewise,  omb’s  Circular  A-11,  Part  3,  Planning  Budgeting  and 
Acquisition  of  Fixed  Assets,  (July  16, 1996),  and  its  Bulletin  No.  95-03, 
Planning  and  Budgeting  for  the  Acquisition  of  Fixed  Assets,  state  that  “the 
planning  for  fixed  asset  acquisitions  should  be  based  on  a  systematic 
analysis  of  expected  benefits  and  costs.” 

However,  even  though  economic  analyses  play  a  critical  role  in  assessing 
whether  system  development  efforts  will  be  cost-effective  and  beneficial. 
Defense  has  not  yet  established  a  set  of  minimum  standards  that  an 
economic  analysis  must  meet  to  be  considered  valid.  In  addition,  it  has  not 
published  official  guidance  for  preparing  an  economic  analysis,  pa&e 
developed  and  published  an  unofficial  economic  analysis  guide  that 
recommended,  but  did  not  require,  standard  methods  and  formats  for 
preparing  an  economic  analysis  for  an  information  system 
development/modemization  project.  According  to  a  pa&e  representative, 
this  unofficial  guide  is  no  longer  adequate  because  it  does  not  require  the 
degree  of  standardization  in  methodology  and  analytic  techniques  needed 
to  support  a  portfolio  management  approach  for  managing  information 
technology  investments,  as  required  by  the  Clinger-Cohen  Act  of  1996.  For 
example,  the  guide  does  not  require  that  returns  on  investment  for  systems 
development/modemization  projects  be  calculated  in  a  standard  manner 
using  a  standard  definition.  This  lack  of  standardization  results  in 
economic  analyses  for  different  systems  that  are  not  comparable  enough 
for  DOD  managers  to  have  a  good  basis  for  deciding  which  systems  offer 
the  highest  payoffs.  The  pa&e  official  stated  that  in  conjunction  with  dod’s 
implementation  of  a  portfolio  management  approach,  it  plans  to  officially 
publish  appropriate  documents  and  provide  minimum  standards  and 
guidance  for  the  economic  analyses  process. 
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Our  review  also  found  that  dod  decisionmakers  do  not  view  economic 
analyses  as  key  tools  for  deciding  whether  to  invest  in  an  information 
system  development  or  modernization  project.  As  a  result,  dod  often  lacks 
complete  and  accurate  information  on  system  development^modemization 
projects’  estimated  costs  and  benefits  at  the  time  decisions  are  made  to 
invest  in  the  projects.  Specifically,  as  the  following  examples  show,  we 
found  that  (1)  economic  analyses  for  many  systems  have  not  been 
submitted  for  independent  review  and  (2)  a  significant  portion  of  those 
that  were  submitted  were  inadequately  prepared.  Thus,  many  of  the 
benefits  that  could  be  derived  from  this  tool  have  not  been  realized. 

•  Twelve  of  the  43  major  migration  systems  have  not  yet  submitted  an 
economic  analysis,  or  an  update  of  a  previously  prepared  economic 
analysis,  for  independent  review.®  These  systems  were  under  development 
or  modernization  and  dod  had  invested  himdreds  of  millions  of  dollars  in 
them  in  total.  These  included  5  systems  that  were  \mder  direct  maisrc 
oversight  and  7  systems  for  which  maisrc  delegated  oversight 
responsibility  to  a  service  or  agency.  These  12  systems,  or  components  of 
them,  are  all  in  the  second  development  phase  or  beyond.®  Delaying  the 
preparation  or  updating  of  a  previously  prepared  economic  analysis  to  the 
later  stages  of  development  for  these  12  systems  defeats  the  purpose  of 
the  economic  analysis,  which  is  to  demonstrate  that  a  proposal  to  invest  in 
a  new  system  is  valid  before  that  investment  is  made. 

•  Four  systems  that  were  under  direct  maisrc  oversight,  were  in  the  first 
development  phase — concept  exploration — and  were,  therefore,  not  yet 
required  by  dod’s  acquisition  regulations  to  submit  an  economic  analysis 
or  an  update  of  a  previously  prepared  economic  analysis  for  independent 
review.  Although  these  four  systems  were  technically  in  compliance  with 
Defense’s  acquisition  regulations,  dod  had  already  made  mjgor 
investments  in  them  without  the  benefit  of  knowing  whether  returns  on 
investment  are  going  to  be  acceptable. 

•  Even  though  Defense  has  not  established  standards  for  the  required 
analyses.  Program  Analysis  and  Evaluation  (pa&e)  staff  who  review  the 
economic  analyses  that  are  under  direct  maisrc  oversight  told  us  that  10  of 


®Of  the  43  major  migration  systems  under  MAISRC  review,  27  were  under  direct  review  by  MAISRC 
and  16  were  delegated  to  other  Defense  oversight  organizations,  with  MAISRC  retaining  responsibility 
for  ensuring  adequate  oversight. 

^e  responsibility  for  independently  reviewing  economic  analyses  for  the  five  systems  that  were 
under  direct  MAISRC  oversight  rests  with  DOD’s  Program  Analysis  and  Evaluation  staff,  while  the 
responsibility  for  reviewing  economic  analyses  for  the  seven  delegated  systems  rests  with  the  various 
Defense  organizations  to  which  MAISRC  delegated  oversight  responsibility. 
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the  19  economic  analyses  reviewed  had  problems. These  problems 
included  the  following: 

•  Understating  or  omitting  the  costs  of  standardizing  data,  implementing 
the  standard  data  in  the  systems,  and  developing  system  interfaces. 

•  Failing  to  estimate  the  amount  of  savings  expected  for  terminating 
duplicative  legacy  systems. 

•  Relying  on  professional  judgment  to  make  imsupported  assumptions 
rather  than  making  objective  analyses  to  estimate  the  value  of  benefits 
and  costs.  For  example,  in  one  case,  the  economic  analysis  estimated  a 
cost  avoidance  associated  with  replacing  an  old  system  by  assuming 
that  both  the  old  and  the  replacement  systems’  software  maintenance 
costs  could  be  accurately  estimated  using  two  different  rates  per  line  of 
code,  but  the  analysis  provided  no  data  supporting  the  validity  of  either 
rate. 

After  identifying  problems  with  these  10  economic  analyses,  pa&e  staff 
worked  with  maisrc  analysts  and  the  systems’  program  managers  to 
address  the  problems.  However,  dod  continued  to  develop  the  systems  in 
spite  of  the  fact  that  the  investments  had  not  been  justified  by  complete 
and  accurate  economic  analyses. 

PA&E  analysts  told  us  that  there  were  other  problems  that  impeded  their 
review  and  verification  of  the  economic  analyses.  For  example,  economic 
analyses  are  often  not  updated  and  provided  to  pa&e  for  review  after  major 
changes  occur  in  the  project,  such  as  significant  cost  grovsdh  or  redirection 
of  the  project.  A  second  problem  is  that  economic  analyses  are  often  not 
supported  by  analyses  of  alternatives  that  weigh  the  cost  and  benefits  of 
various  technical  options,  such  as  whether  to  buy  commercial  off-the-shelf 
software  or  develop  a  system  in-house.  “  This  analysis  would  help  Defense 
decisionmcikers  make  sound  decisions  on  whether  the  proposed 
alternatives  offer  sufficient  military  or  economic  benefits  to  be  worth  their 
cost,  and  to  determine  which  alternative  is  the  best  approach.  It  would 
also  identify  alternatives  that  dod  may  want  to  reconsider  at  a  later  time  if 
the  selected  approach  runs  into  difficulties. 


did  not  identify  concerns  or  problems  with  the  economic  analyses  for  the  remaining  8  of  the  43 
major  migration  systems  that  were  under  direct  MAISRC  oversight  for  which  MAISRC  had  delegated 
oversight  responsibility  to  another  DOD  organization. 

“The  analysis  of  technical  alternatives  would  normally  precede  the  economic  analysis.  In  Defense,  an 
economic  analysis  weighs  the  costs,  benefits,  and  risks  associated  with  maintaining  the  status  quo 
versus  the  chosen  technical  solution. 
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DOD’s  Acquisition 
Oversight  Organizations 
Lack  Assurance  That  Major 
Systems  Comply  With 
Technical  and  Data 
Standards 


Defense  has  established  several  sets  of  standards  that  are  designed  to 
ensure  that  systems  developed  are  compatible  with  its  communications 
and  computing  infrastructme  and  that  they  are  technically  interoperable 
with  each  other.  Some  Defense  leaders  consider  systems  interoperability 
and  the  ability  to  exchange  data  across  frmctional  lines  to  be  the  most 
important  consideration  in  migration  system  development,  transcending 
economic  benefits.  These  standards  include  the  Technical  Architecture 
Framework  for  Information  Management  (tafim),  Defense  Information 
Infrastructure  Common  Operating  Environment  (on  coe),  and  dod 
standard  data.^^  dod  system  acquisition  directives  call  on  maisrc  and  dab  to 
ensure  that  program  managers  comply  with  the  Department’s  policies  and 
procedures  and  use  best  practices  in  developing  and  modernizing 
individual  information  systems.  These  best  practices  include  building 
systems  and  databases  that  comply  with  applicable  technical  standards 
and  use  dod  standard  data. 


However,  maisrc  and  dab  do  not  have  adequate  assurance  that  the  major 
migration  systems  are  complying  with  applicable  technical  standards*®  and 
are  using  standard  data.  For  example,  out  of  43  major  systems  under 
MAISRC  oversight,  program  managers  reported  to  maisrc  that  (1)  only  19 
were  in  compliance  with  the  tafim  standards  or  had  plans  to  comply  with 
the  TAFIM  standards  and  (2)  9  systems  were  using  dod  standard  data  or  had 
plans  to  use  standard  data.*^  We  found  similar  results  for  compliance  with 
Dn  COE  standards.  Specifically,  program  managers  reported  that  only  25 
systems  were  in  compliance  or  had  plans  to  be  in  compliance  with  Dn  coe. 
These  self-reports  by  program  managers  are  questionable  because  they  are 
not  independently  verified. 

In  addition  to  obtaining  information  on  technical  and  data  standards 
directly  from  program  managers,  maisrc  and  dab  also  obtain  such 
information  from  the  Defense  Information  Systems  Agency  (disa).  disa 
supports  MAISRC  and  dab  oversight  of  major  systems  by  performing 
technical  reviews  of  system  documentation  to  determine  if  it  indicates  that 
interoperability  issues  are  being  addressed  or  will  be  addressed  as  the 


^^See  footnote  3  for  more  information  on  technical  and  data  standards. 

^^All  DOD  technical  standards  may  not  apply  to  all  systems  and  initiatives.  For  example,  those 
standards  that  apply  to  application  software  and  data  would  not  apply  to  a  system  or  initiative  that 
included  only  computer  equipment  or  hardware.  To  illustrate,  Defense’s  High  Performance  Computing 
Modernization  Program  initiative  largely  involves  the  purchase  of  computer  equipment  and  not  the 
development  of  software.  Therefore,  application  software-related  standards  may  not  apply. 

^^In  addition,  information  available  to  DAB  indicated  that  all  six  migration  systems  under  DAB 
program  managers  have  already  prepared  plans  to  bring  their  systems  into  compliance  with  DOD 
technical  standards. 
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system  is  developed.  According  to  a  disa  representative,  disa  reviewed 
system  documentation  that  had  been  prepared  for  37  of  the  43  maisrc 
migration  systems  and  all  6  of  the  dab  systems  and  found  that  the 
dociunentation  indicated  interoperability  issues  were  planned  to  be 
addressed  for  each  of  them. 

DISA  also  provides  feedback,  advice,  and  assistance  on  interoperability  and 
related  issues  during  integrated  product  team  meetings  with  maisrc,  dab, 
and  system  program  managers.  During  these  meetings  and  on  other 
occasions  when  it  is  asked  to  do  so,  disa  provides  maisrc  and  dab  general 
information  on  compliance  with  standards,  such  as  whether  the  major 
systems’  program  managers  have  contacted  disa  and  appear  to  be  making 
reasonable  attempts  to  bring  their  systems  into  compliance  with 
applicable  standards,  disa  also  assists  program  managers  in  developing 
strategies  and  cost  estimates  for  achieving  compliance  with  standards. 

However,  disa  does  not  regularly  report  detailed  information  to  maisrc  and 
DAB  that  would  enable  these  oversight  organizations  to  ensure  that  each 
major  system  is  adequately  complying  with  applicable  technical  and  data 
standards.^®  For  example,  disa  does  not  regularly  report  to  maisrc  and  dab 
such  detailed  information  as  (1)  each  system’s  current  compliance  with 
applicable  standards — ^that  is,  each  system’s  current  status  relative  to  the 
eight  levels  that  dod  has  defined  for  the  dh  coe,  and  each  system’s  number 
and  percentage  of  data  elements  that  have  been  approved  as  dod  data 
standards  in  the  Defense  Data  Dictionary  System,  (2)  whether  each 
program  office  prepared  sound  strategies,  schedules,  and  cost  estimates 
for  achieving  compliance  with  applicable  standards,  (3)  whether  each 
system’s  compliance  strategy  and  cost  estimate  were  approved  by  disa, 

(4)  each  system’s  current  schedule  and  cost  status  for  achieving 
compliance  compared  with  its  baseline  schedule  and  cost  estimate,  and 

(5)  whether  each  system  has  been  independently  certified  by  disa’s  Joint 
Interoperability  Testing  Command  to  be  in  compliance  with  the  technical 
and  data  standards,  maisrc  analysts  responsible  for  overseeing  the  major 
migration  systems  confirmed  that  they  did  not  know  many  of  the  systems’ 
current  status  regarding  compliance  with  applicable  technical  standards 
and  use  of  dod  standard  data,  or  whether  the  program  managers  had 


DISA  representative  stated  that  program  managers  are  supposed  to  input  data  into  DOD’s  DIST 
database  that  could  be  used  by  MAISRC  and  DAB  to  track  each  individual  system’s  progress  in 
complying  with  on  coe.  For  example,  the  representative  noted  that  program  managers  are  supposed  to 
input  compliance  data  into  DIST,  including  (1)  their  systems’  current  level  of  compliance  with  on  coe, 
(2)  information  on  their  strategies  for  achieving  compliance,  and  (3)  the  expected  date  and  cost  to 
achieve  compliance.  However,  as  discussed  in  appendix  IV,  our  analysis  of  the  data  in  the  DIST 
database  showed  they  are  incomplete  and  inaccurate.  Until  DOD  corrects  these  deficiencies,  MAISRC 
and  DAB  cannot  rely  on  DIST  for  tracking  compliance  with  standards  for  individual  systems. 
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developed  and  were  following  sound  strategies  for  bringing  the  systems 
into  compliance. 

The  technical  and  data  standards  are  supposed  to  help  pave  the  way  to  an 
interoperable  systems  environment.  However,  without  complete  and 
accurate  data  on  individual  systems’  compliance  with  standards,  maisrc 
and  DAB  cannot  assure  Defense’s  Chief  Information  Officer  that  the 
Depaotment’s  major  information  systems  are  complying  with  the 
standards.  Without  this  information,  as  well  as  standards  comphance 
information  from  the  managers  of  dod’s  nonmajor  systems,  the  cio  cannot 
gauge  the  Department’s  progress  toward  achieving  its  goal  of  an 
interoperable  systems  environment.  Additionally,  this  information  is 
critical  if  the  cio  is  to  successfully  implement  an  integrated  technical 
architecture  for  the  Department  as  required  by  the  Clinger-Cohen  Act  of 
1996. 


Better  Oversight 
Could  Have  Helped  to 
Prevent  Problems 
Identified  in  Previous 
GAO  Reviews 


The  lack  of  rigorous  oversight  for  the  migration  strategy  has  increased  the 
risk  that  Defense  will  pursue  flawed  system  strategies.  In  fact,  our 
previous  reviews  of  migration  systems  identified  a  number  of  problems 
that  could  have  been  prevented  had  there  been  better  oversight  by  maisrc 
and  the  asd  C3I.  For  example,  in  several  reviews,  we  found  that  functional 
areas  did  not  account  for  various  categories  of  significant  costs  when 
making  their  migration  decisions.  These  findings  are  highhghted  as 
follows. 


•  In  reviewing  the  depot  maintenance  standard  system  migration  effort,  we 
found  that  Defense  did  not  address  the  full  costs  of  developing  interfaces 
needed  to  allow  system  components  to  exchange  data  with  information 
systems  currently  used  by  the  services  to  accomplish  their  missions.  One 
official  estimated  that  this  represented  $70  million  in  costs.^® 

•  In  analyzing  what  it  would  cost  to  develop  its  Standard  Accounting  and 
Reporting  System  (stars),  we  reported  that  Defense  neglected  to  consider 
internal  project  management  costs  and  costs  to  enhance  all  of  the  stars 
components  to  bring  them  into  comphance  with  dod’s  standard  general 
ledger,  key  accounting  requirements,  and  the  standard  budget  and 
accounting  classification  code.^^ 


^‘^Defense  Management:  Selection  of  Depot  Maintenance  Standard  System  Not  Based  on  Sufficient 
Analyses  (GAO/AIMD-95-110,  July  13,  1995). 

^^DOD  Accounting  Systems:  Efforts  to  Improve  System  for  Navy  Need  Overall  Structure 
(GAO/AIMD-96-99,  September  30, 1996). 
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•  In  reviewing  the  transportation  migration  effort,  we  found  that  Defense 
did  not  include  all  costs  associated  with  its  evaluation  of  in-house  systems 
when  analyzing  costs  and  savings  for  its  28  migration  systems.  These 
included  $16  million  for  its  analysis  of  candidate  migration  systems  and 
$2  million  for  maintaining  migration  system  hardware.  We  also  found  that 
if  these  costs  were  included  in  its  systems  selection  analyses,  Defense 
would  have  found  that  the  overall  return  on  investment  would  have 
decreased  and  that  it  may  actually  lose  money  on  its  investment.^® 

In  previous  reviews,  we  also  found  that  Defense  functional  areas  did  not 
adequately  consider  other  alternatives  to  developing  systems  in-house.  For 
example,  while  the  transportation  area  reviewed  commercial  off-the-shelf 
transportation  software  projects  for  some  transportation  business  areas, 
this  review  was  inadequate  because  it  did  not  (1)  analyze  the  degree  to 
which  unmodified  software  could  meet  unique  Defense  requirements, 

(2)  identify  the  expected  cost  to  make  necessary  software  modifications, 

(3)  determine  the  time  required  to  make  the  modifications,  and  (4)  provide 
for  a  hands-on  view  of  the  software  in  operation.  In  addition.  Defense 
concluded  that  software  packages  that  could  provide  some  degree  of 
transportation  functionality  would  require  modifications  that  were  too 
costly.  However,  Defense  could  not  provide  documented  analysis  to 
support  this  conclusion.  Further,  Defense  planned  on  making  $13  million 
worth  of  software  modifications  to  just  five  of  its  in-house  selections.  We 
believe  better  oversight  by  the  cio  (then  referred  to  as  the  Senior 
Information  Management  Official)  may  well  have  forced  greater 
consideration  of  commercial  packages  in  the  transportation  area. 

Finally,  our  review^®  of  Defense’s  effort  to  develop  a  standard  suite  of 
migration  systems  for  materiel  management  showed  that  the  Department 
spent  hundreds  of  nulhons  of  dollars  without  achieving  the  expected 
benefits  because  it  did  not  adequately  anticipate  and  mitigate  risks.  From 
1992  to  late  1995,  Defense  spent  about  $714  million  developing  standard 
systems  with  minimal  results.  During  that  time,  there  were  dramatic 
changes  in  the  goals  and  expectations  for  the  program  and  only  one 
application  was  partially  deployed.  Because  of  changes  in  objectives  and 
scheduling  and  problems  in  development,  prospects  for  achieving  the 
original  objective  of  implementing  a  standard  suite  of  integrated  materiel 
management  systems  appeared  dim. 


‘^Defense  Transportation:  Migration  Systems  Selected  Without  Adequate  Analysis  (GAO/AIMD-96-81, 
August  29, 1996). 

^^Defense  IRM:  Critical  Risks  Facing  New  Materiel  Management  Strategy  (GAO/AIMD-96-109, 
September  6, 1996). 
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In  1996,  Defense  finally  abandoned  its  strategy  to  develop  a  standard  suite 
of  materiel  management  systems  because  of  funding  cuts,  cost  overruns, 
and  schedule  delays  and  embarked  on  a  new  strategy  that  involved 
individual  deployment  of  nine  system  applications  at  selected  sites  as  the 
applications  were  developed.  However,  the  decision  to  drastically  change 
the  course  of  the  strategy  was  initiated  without  first  conducting  critical 
economic  and  risk  assessments  that  would  estimate  the  costs,  benefits, 
and  risks  of  alternative  strategies  and  having  the  analyses  independently 
reviewed  by  maisrc  and  pa&e.^®  The  need  for  department-level  review  of 
analyses  supporting  this  decision  was  important,  given  the  fact  that  the 
new  strategy  represented  a  departure  from  Defense’s  goal  of  eliminating 
redimdant  legacy  systems  and  varied  business  processes.^^ 


Defense  Lacks 
Visibility  Over 
Migration  Effort 


A  successful  information  technology  investment  process  cannot  operate 
without  accurate,  reliable,  and  up-to-date  data  on  project  costs,  benefits, 
and  risks.  It  is  the  basis  for  informed  decision-making.  However,  in 
implementing  the  migration  strategy.  Defense  neglected  to  provide 
visibility  over  progress  and  costs.  The  absence  of  reliable  and  accurate 
performance,  cost,  and  schedule  information  on  the  migration  effort  has 
been  debilitating  in  several  respects.  First,  it  has  impaired  the 
Department’s  ability  to  demonstrate  whether  the  migration  strategy  has 
been  successful.  Second,  it  has  kept  Defense  from  providing  the  Congress^^ 
with  accurate  and  reliable  information  needed  for  oversight  purposes. 
Third,  it  has  prevented  Defense  from  providing  its  own  senior  managers 
with  information  needed  to  oversee  the  migration  effort. 


Key  Performance  Issues  Since  it  embarked  on  the  migration  strategy.  Defense  has  not  tracked 

Have  Not  Been  Tracked  overall  departmentwide  savings  or  validated  improvements  to  operations 

resulting  from  the  migration  strategy.  In  particular,  it  has  not  been 
systematically  tracking  such  key  performance  issues  as  (1)  administrative 
and  operational  cost  savings  resulting  from  elimination  of  redundant 


-^e  materiel  management  standard  system  was  originally  projected  to  cost  $5.3  billion  to  develop 
and  deploy.  The  threshold  for  major  information  system  projects  is  $360  million  in  total  life-cycle 
costs. 

-^Under  the  revised  strategy,  the  military  services  would  be  allowed  to  keep  their  legacy  systems 
longer  than  anticipated,  and  some  legacy  systems  would  not  be  shut  down. 

-For  example,  Defense  reported  to  the  Congress  that  it  had  selected  363  migration  and  interim 
systems,  using  DIST  as  the  source  of  the  data.  Defense  also  provided  cost  data  that  were  collected  by 
the  Office  of  the  ASD  C3I  based  on  a  different  list  of  245  migration  and  interim  systems.  Since  the  ASD 
C3rs  staff  relied  largely  on  an  official  list  of  245  systems  that  ASD  C3I  issued  in  July  1995  to  provide 
cost  information  to  the  Congress,  that  information  was  incomplete.  In  addition,  the  ASD  C3rs  staff  did 
not  reconcile  the  two  lists  of  systems  in  its  report  to  the  Congress. 
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systems,  (2)  cost  reductions  resulting  from  improved  information  systems 
support  to  functional  areas,  (3)  management  or  staff  productivity 
improvements,  and  (4)  benefits  accruing  to  mission  effectiveness  that  are 
attributable  to  information  technology  support.  Having  this  type  of 
information  is  the  only  means  of  ensuring  that  the  billions  of  dollars  being 
spent  on  the  migration  are  producing  sufficient  returns  and  achieving 
departmentwide  progress.^  Without  it.  Defense  cannot  justify  whether  the 
migration  strategy  has  been  a  worthwhile  investment  or  support  the  need 
to  continue  pursuing  migration. 

At  one  point  in  the  migration  effort,  as  directed  by  the  Congress,^'*  Defense 
developed  performance  measures.  However,  it  did  not  regularly  coUect 
data  on  all  the  measures,  and  the  accuracy  of  much  of  the  information  it 
collected  was  questionable  because  of  the  data  integrity  problems 
plaguing  the  database  Defense  relies  on  for  migration  inventory  and 
schedule-related  information.  In  1995,  the  House  Committee  on  National 
Security  directed  Defense^^  to  reevaluate  its  measures.  In  March  of  1997,  a 
Defense  working  group  proposed  a  revised  set  of  performance  measures 
relating  to  migration  systems  and  other  information  technology  issues 
within  the  Department.  However,  senior  Defense  management  did  not 
approve  the  measures  and  instead  tasked  the  working  group  to  redo  them 
to  ensure  that  they  are  linked  to  the  Department’s  strategic  information 
technology  management  goals.  The  working  group  has  not  established  a 
schedule  for  completing  this  effort  and  finalizing  the  measures. 

Having  performance  measures  wdll  help  to  validate  individual  reports  of 
migration  successes,  such  as  the  migration  systems  characterized  as 
successful  in  Defense’s  report  to  the  Congress  pursuant  to  Section  381  of 
the  National  Defense  Authorization  Act  for  Fiscal  Year  1995.  These  are  the 
Standard  Procurement  System,  the  Distribution  Standard  System,  the 
Defense  Civilian  Personnel  Data  System,  and  the  Defense  Medical 
Logistics  Standard  Support  System. 


According  to  a  DOD  representative,  the  Department  performed  intense  reviews  of  migration  systems 
during  the  last  three  cycles  of  its  planning,  programming,  and  budgeting  process.  During  these  reviews, 
DOD  asked  program  managers  and  other  senior  managers  to  provide  performance  information  on  their 
systems.  However,  the  representative  stated  that  the  performance  information  provided  was  limited 
because  DOD  does  not  routinely  track  this  information  for  its  information  systems. 

“^National  Defense  Authorization  Act  for  Fiscal  Year  1995. 

Rep.  No.  104-131,  p.  161. 
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Cost  Information  Is 
Incomplete  and  May  Be 
Significantly  Understated 


We  could  not  determine  the  full  cost  of  the  migration  effort  because 
Defense’s  migration  cost  information  is  incomplete  and  may  be 
significantly  understated.  For  example,  when  Defense  provided  migration 
cost  data  to  us  in  December  1996,  it  could  only  provide  costs  for  242  of  363 
migration  systems.  We  did  not  include  costs  in  this  report  for  27  of  242 
systems  for  which  dod  collected  costs  because  the  systems  were 
classified.  However,  the  remaining  121  systems  were  not  included  because 
the  Office  of  the  asd  C3i  had  not  collected  the  costs  for  these  systems. 
When  we  obtained  cost  information  directly  from  three  of  the  functional 
areas,  we  found  that  at  least  $1.6  billion  in  costs  had  been  excluded  from 
the  approximately  $18  billion  total.  This  included  about  $1.4  billion  for 
clinical  health  systems,  $56  million  for  civilian  personnel  systems,  and 
$111  million  for  transportation  systems. 

We  also  found  that  the  $18  bilhon  total  cost  estimate  does  not  account  for 
some  very  important  costs  related  to  developing,  deploying,  and 
maintaining  migration  systems  both  before  and  after  a  project  has  been 
initiated.  For  example,  the  transportation  functional  area  has  an  office 
designated  to  oversee  the  development  and  deployment  of  its  migration 
systems.  But,  when  accounting  for  costs  for  the  systems,  dod  does  not 
factor  in  the  cost  to  maintain  this  oversight  responsibility.  In  addition,  our 
previous  reviews  of  finance  and  logistics  migration  efforts  have  found  that 
DOD  did  not  account  for  costs  for  these  projects  relating  to  such  activities 
as  project  management  and  developing  interfaces  with  other  systems.  Our 
detailed  analysis  of  dod’s  migration  cost  information  is  provided  in 
appendix  III. 


Schedule  Information  Is  We  could  not  accurately  determine  how  many  legacy  systems  have  been 

Unreliable  terminated  and  how  many  are  scheduled  for  termination  because  the 

database  Defense  uses  to  track  information  systems  is  plagued  with  data 
integrity  problems. 

Defense’s  own  analyses  of  the  database  have  shown  that  it  cannot  readily 
provide  simple  descriptive  information  for  many  systems.  For  example,  a 
February  1997  dod  analysis  of  dist  showed  that  55  percent  of  the  migration 
systems  in  the  database  had  incomplete  information  on  interfaces  with 
other  systems  and  77  percent  had  incomplete  data  on  installations  where 
the  systems  operated. 

Our  analysis  of  dist  found  that  61  percent  the  migration  system 
implementation  dates  and  32  percent  of  legacy  system  termination  dates 
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were  questionable.  In  addition,  when  we  compared  dist  scheduling 
information  to  information  maintained  by  three  functional  areas  we 
reviewed,  we  found  that  most  of  the  migration  systems  implementation 
dates  and  legacy  system  termination  dates  in  dist  were  incorrect.  For 
example,  dist  showed  that  92  legacy  systems  were  terminated  by 
April  1996  in  the  clinical  health,  civilian  personnel,  and  transportation 
areas,  while  functional  area  managers  told  us  that  only  43  had  actually 
been  terminated.  Also,  for  the  three  functional  areas,  dist  showed  that  53 
legacy  systems  were  scheduled  for  future  termination  while  functional 
area  managers  told  us  that  91  were  slated  for  future  termination. 

We  also  found  that  Defense  has  not  ensured  that  the  data  definitions  and 
formats  used  in  dist  are  fully  compatible  with  data  maintained  in  other 
Defense  information  systems  that  track  and  report  on  systems.  While  dod 
is  attempting  to  address  this  issue,  disa  provided  us  information  showing 
that  only  66  of  the  218  data  elements  contained  in  dist  have  been  approved 
as  standard  data  elements  in  the  Defense  Data  Dictionary  System.  Without 
standard  definitions  and  data  formats,  data  cannot  be  easily  transferred  to 
DIST  from  the  other  systems  that  may  be  used  by  functional  area  managers 
and  other  decisionmakers. 

Defense  officials  acknowledge  that  dist  is  incomplete  and  inaccurate,  and 
the  Department  has  begun  efforts  to  make  the  database  more  accurate  and 
more  user  friendly.  However,  Defense  officials  also  stated  that  they  still 
used  dist  to  generate  reports  to  the  Congress  because  it  was  the  only 
departmentwide  database  containing  schedule  and  other  descriptive  data 
on  all  Defense  migration  and  legacy  systems.  A  detailed  analysis  of  dist’s 
integrity  problems  is  provided  in  appendix  IV. 
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Defense  has  begun  implementing  Division  E  of  the  Clinger-Cohen  Act  of 
1996  (Public  Law  104-106),  which  the  Congress  passed  in  an  effort  to  put 
an  end  to  problems  associated  with  the  development  and  implementation 
of  goverrmient  information  systems — ^such  as  those  evident  in  the 
migration  effort.  Under  this  legislation,  Defense  is  required  to  establish 
additional  controls  to  ensure  that  more  attention  is  devoted  to  the 
selection,  control,  and  evaluation  of  information  technology  projects  and 
that  the  Department’s  technology  investments  are  managed  from  a 
portfolio  perspective. 

Implementing  the  Clinger-Cohen  Act  in  dod  can  bring  meaningful  reform 
to  the  management  of  migration  and  other  information  technology 
investments.  However,  the  current  structure  of  the  Chief  Information 
Officer  (cio)  position  in  the  Department  will  not  permit  the  cio  to  devote 
full  attention  to  reforming  information  resources  management  within  dod. 
And  it  wiU  continue  to  impose  responsibilities  on  the  cio  that  conflict  with 
the  cio’s  obligation  to  provide  independent  oversight  over  the  development 
and  implementation  of  information  systems. 


The  Clinger-Cohen 
Act 


The  Clinger-Cohen  Act  recognizes  that  the  key  to  successfully  managing 
information  technology  projects  is  ensuring  that  investment  processes 
provide  for  the  continued  identification,  selection,  control,  life-cycle 
management,  and  evaluation  of  information  technology  investments.  Best 
practices  on  which  the  act  is  based  have  shovm  that  to  ensure  an 
investment  process  is  successful,  top  executives  need  to  periodically 
assess  all  major  projects — ^proposed,  under  development,  and 
operational — then  prioritize  them  and  make  funding  decisions  based  on 
factors  such  as  cost,  risk,  return  on  investment,  and  support  of 
mission-related  outcomes.  Once  projects  are  selected  for  funding, 
executives  need  to  monitor  them  continually,  taking  quick  actions  to 
resolve  development  problems  and  mitigate  risks.  After  a  project  is 
implemented,  executives  should  evaluate  actual  versus  expected  results 
and  revise  their  investment  management  process  based  on  lessons 
learned. 

As  our  guide.  Assessing  Risks  and  Returns:  A  Guide  for  Evaluating  Federal 
Agencies’  Information  Technology  Investment  Decision-making,^  points 
out,  a  key  to  success  in  this  type  of  management  is  considering  aU  the 
major  technology  investments  that  are  vying  for  funding  at  a  designated 
level  (departmental,  functional  area,  or  service/agency  level)  as  a  total 


‘GAO/AIMD-IO.1.13  February  1997,  Version  1. 
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package,  ox  portfolio,  of  possible  technology  investments.  Once  this 
perspective  is  adopted,  an  organization  can  focus  scarce  information 
technology  resources  on  the  projects  with  the  greatest  impact  on  mission 
and  concentrate  management  attention  on  those  high-impact  projects  that 
become  troubled  projects.  It  can  also  establish  performance  goals  and 
stop  or  replace  those  systems  or  initiatives  that  fail  to  meet  those  goals. 

This  type  of  decision-making  process  can  be  applied  to  almost  any 
organization,  even  one  that  is  as  highly  decentralized  as  dod.  According  to 
our  investment  guide,  separate  information  technology  investment 
decision-making  processes  can  exist  at  various  levels.  In  dod,  such 
processes  could  exist  at  the  departmental  level  and  the  functional  area  or 
service/agency  level,  provided  that  the  Department  can  identify  which 
major  projects  and  initiatives  should  be  managed  at  each  level.  Criteria  for 
determining  projects  that  should  be  managed  at  the  various  levels  may 
include  the  dollar  amount  of  the  investment,  the  degree  of  risk  associated 
with  the  project,  and  whether  the  system  is  to  be  shared  across  functional 
area  lines  or  service/agency  lines. 

The  Clinger-Cohen  Act  contains  the  following  additional  requirements  that 
are  important  for  dod  to  implement  in  order  to  address  problems  evident 
in  its  migration  strategy. 

•  Agencies  are  to  determine  whether  their  administrative  and 
mission-related  business  processes  should  be  improved  before  investing  in 
major  information  systems  to  support  them. 

•  The  investment  process  is  to  provide  a  means  for  senior  management  to 
obtain  timely  information  regarding  progress  (at  established  milestones)  in 
terms  of  cost,  capability  of  the  system  to  meet  requirements,  timeliness, 
and  quality.  It  should  also  provide  for  the  evaluation  of  the  results  of 
information  technology  investments. 

•  Performance  measures  are  to  be  prescribed  for  information  technology 
used  by  or  to  be  acquired  for  the  agency. 

•  The  CIO  is  to  monitor  the  performance  of  information  technology 
programs;  evaluate  the  performance  of  those  programs  on  the  basis  of 
applicable  performance  measures;  and  advise  the  agency  head  regarding 
whether  to  continue,  modify,  or  terminate  the  program  or  project. 

•  The  CIO  is  to  be  responsible  for  providing  advice  and  other  assistance  to 
agency  heads  and  senior  managers  to  ensure  that  information  technology 
is  acquired  and  information  resources  are  managed  for  the  agency  in  a 
manner  that  implements  the  policies  and  procedures  of  the  act  and  the 
priorities  of  the  agency  head. 
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.  The  CIO  is  to  develop,  maintain,  and  facilitate  the  implementation  of  a 
soimd  and  integrated  information  technology  architecture  for  the  agency. 
The  architecture  is  an  integrated  framework  for  evolving  or  maintaining 
existing  information  technology  and  acquiring  new  information  technology 
to  achieve  the  agency’s  strategic  and  information  resources  management 
(irm)  goals. 


DOD  Implementation 
of  the  Clinger-Cohen 
Act 


As  a  first  step  in  implementing  the  Clinger-Cohen  Act,  on  June  2, 1997,  the 
Secretary  of  Defense  outlined  his  expectations  for  improvements  in 
information  technology-related  management  processes  and  information 
resources.^  We  believe  the  expectations  identified  by  the  Secretary  are  an 
excellent  starting  point  for  implementing  the  Clinger-Cohen  Act  and 
bringing  meaningful  change  to  the  current  information  technology 
management  process.  For  example,  the  Secretary  has  called  on  the  cio  to 
design  and  implement  a  process  for  maximizing  the  value  and  assessing 
and  managing  the  risks  of  dod  information  technology  acquisitions.  This 
process  is  to 


•  provide  for  the  selection  of  information  technology  investments  to  be 
made  by  the  Department,  the  management  of  such  investments,  and  the 
evaluation  of  the  results  of  such  investments; 

•  be  integrated  with  processes  for  making  budget,  financial,  and  program 
management  decisions; 

•  include  minimum  criteria  to  be  applied  in  considering  whether  to 
undertake  a  particular  investment  in  information  systems,  including 
criteria  related  to  the  quantitatively  expressed  projected  net,  risk-adjusted 
return  on  investment,  and  specific  quantitative  and  qualitative  criteria  for 
comparing  and  prioritizing  alternative  information  system  investment 
projects; 

•  identify,  for  each  proposed  investment,  quantifiable  measurements  for 
determining  the  net  benefits  and  risks  of  the  investment;  and 

•  provide  the  means  for  senior  managers  to  be  able  to  obtain  timely 
information  regarding  the  progress  of  an  investment  m  an  information 
system,  including  the  milestones  for  measuring  progress  on  an 
independently  verifiable  basis,  in  terms  of  cost,  capability  of  the  system  to 
meet  specified  requirements,  timeliness,  and  quality. 


In  addition,  the  Secretary  has  called  on  the  cio  to  institutionalize 
performance-based  and  results-based  management  for  information 


^Memorandum  from  the  Secretary  of  Defense  to  the  Secretaries  of  the  Military  Departments,  Chairman 
of  the  Joint  Chiefs  of  Staff,  Under  Secretaries  of  Defense,  and  others  on  the  Implementation  of 
Subdivision  E  of  the  Clinger-Cohen  Act  of  1996,  dated  June  2, 1997. 
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technology.  In  doing  so,  the  cio  is  to  work  with  the  Chief  Financial  Officer, 
the  Principal  Staff  Assistants  (psas),  and  the  Defense  components.  The  cio 
is  to  establish  goals  for  improving  the  efficiency  and  effectiveness  of  dod 
operations  and  issue  instructions  to  functional  areas  on  performance 
measurements.  The  cio  is  also  to  monitor  the  performance  of  information 
technology  programs,  evaluate  the  performance  of  those  programs  on  the 
basis  of  applicable  performance  measurements,  and  advise  the  Secretary 
of  Defense  regarding  whether  to  continue,  modify,  or  terminate  programs 
or  projects.  Additionally,  the  Secretary  established  a  Chief  Information 
Officer  Coimcil  for  dod  to  serve  as  the  principal  forum  for  discussing 
improvements  in  dod  practices  for  the  management  of  information 
technology. 

In  outlining  his  expectations,  the  Secretary  noted  that  the  act  poses 
questions  that  should  be  answered  before  investing  in  information 
technology,  including: 

•  What  functions  are  we  performing  and  are  they  consistent  with  the 
mission? 

•  If  we  should  be  performing  particular  functions,  could  they  be  performed 
more  effectively  and  at  lower  cost  by  the  private  sector? 

The  Secretary  further  stated  that  if  a  function  should  indeed  be  performed 
by  the  Department,  the  law  requires  that  the  function  be  examined  and 
redesigned  or  reengineered  before  applying  new  technology. 


Challenges 
Confronting  DOD  in 
Implementing 
Clinger-Cohen 


Proper  implementation  of  the  Clinger-Cohen  Act  would  help  to  address  a 
number  of  weaknesses  that  are  currently  standing  in  the  way  of  Defense’s 
ability  to  provide  a  good  decision-making  and  oversight  environment  for 
information  technology  projects.  For  example,  with  full  implementation  of 
the  act.  Defense  could  begin  considering  information  technology 
investments  as  a  total  package  of  possible  projects  so  that  it  can  target 
resources  to  those  projects  having  the  greatest  impact  on  mission  and 
concentrate  management  attention  on  troubled  areas.  It  could  also 
strengthen  visibility  over  project  performance,  costs,  and  schedules  so 
that  senior  managers  can  begin  comparing  the  results  being  achieved 
against  projected,  costs,  benefits,  and  risks  and  to  identify  actual  or 
potential  managerial,  organizational,  or  technical  problems. 

Moreover,  in  implementing  the  act,  Defense  has  an  opportunity  to 
(1)  begin  enforcing  compliance  with  data  and  technical  standards  to 
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ensxire  that  dod’s  goals  for  interoperability  and  the  sharing  of  information 
are  met  and  (2)  increase  oversight  for  functional  area  assessments  of 
whether  to  reengineer,  migrate,  or  undertake  some  other  path  toward 
improvement. 

However,  the  current  structure  of  the  cio  position  in  Defense  will  not 
permit  the  cio  to  effectively  serve  as  a  bridge  between  top  management, 
line  management,  and  information  management  support  officials  and 
identify  opportunities  to  use  information  technology  to  enhance 
performance. 

The  Chnger-Cohen  Act  requires  that  information  resources  management 
be  the  cio’s  primary  responsibility  and  that  the  cio  be  involved  in  key 
decisions  regarding  the  application  of  information  technology  in  support 
of  the  agency’s  missions.  Currently,  the  Assistant  Secretary  of  Defense  for 
Command,  Control,  Conununications  and  Intelligence  (asd  C3I)  also  acts 
as  the  CIO.  By  asking  the  cio  to  also  shoulder  a  heavy  load  of  programmatic 
responsibility.  Defense  has  made  it  difficult,  if  not  impossible,  for  the  cio 
to  devote  full  attention  to  irm  issues.  These  issues  go  well  beyond  the 
development  and  modernization  of  information  systems. 

For  example.  Defense  needs  its  cio  to  be  heavily  involved  with 
implementing  a  more  aggressive  and  proactive  computer  security 
program.  As  we  reported  in  May  1996,^  attackers  have  seized  control  of 
entire  Defense  systems,  many  of  which  support  critical  functions,  such  as 
weapons  systems  research  and  development,  logistics,  and  finance. 
Attackers  have  also  stolen,  modified,  and  destroyed  data  and  software.  In 
addition.  Defense  needs  its  cio  to  help  ensure  that  Year  2000'*  corrections 
are  made  to  all  of  dod’s  information  systems.  If  systems  are  not  corrected 
on  time,  the  impact  on  Defense  operations  could  be  widespread,  costly, 
and  debilitating  to  important  warfighting  and  administrative  operations. 
While  DOD  has  delegated  year  2000  responsibility  to  its  components,  it  still 
needs  to  ensure,  at  the  department  level,  that  sufficient  priority  and 


^Information  Security:  Computer  Attacks  at  Department  of  Defense  Pose  Increasing  Risks 
(GAO/AIMD-96-84,  May  22, 1996). 

^The  Year  2000  problem  is  rooted  in  the  way  dates  are  recorded  and  computed  in  automated 
information  systems.  For  the  past  several  decades,  systems  have  typically  used  two  digits  to  represent 
the  year  in  order  to  conserve  on  electronic  data  storage  and  reduce  operating  costs.  With  this  two-digit 
format,  however,  the  year  2000  is  indistinguishable  from  1900, 2001  from  1901,  etc.  As  a  result  of  this 
ambiguity,  system  or  application  programs  that  use  dates  to  perform  calculations,  comparisons,  or 
sorting  may  generate  incorrect  results  when  working  with  years  after  1999. 
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resources  are  being  devoted  to  the  problem  and  that  all  systems  have  been 
identified  and  corrected.^ 

There  is  also  a  direct  conflict  of  responsibilites  between  the  oversight  and 
programmatic  obligations  associated  with  the  two  positions.  The  asd  C3i 
serves  as  the  Principal  Staff  Assistant  for  command,  control, 
communications,  and  intelhgence  systems.  These  systems  represent  about 
45  percent  of  the  migration  system  investment — ^about  $8.5  bilUon  of  the 
total  $18  bilUon  migration  investment.  This  poses  a  conflict  for  both 
control  mechanisms  we  assessed.  For  the  first  control — ^the  asd  C3i 
approval  process — ^the  same  individual  is  responsible  for  both  selecting  a 
system  and  approving  the  selection.  For  the  second  control — ^acquisition 
oversight — Defense’s  Acquisition  Executive  is  also  responsible  for 
developing  14  of  the  43  major  automated  information  systems  imder  the 
Major  Automated  Information  System  Review  Council  (maisrc).® 

A  second  challenge  confronting  dod  in  implementing  the  Clinger-Cohen 
Act  is  the  prevailing  organizational  structure  and  embedded  culture  found 
throughout  the  Department.  Specifically,  the  three  military  services  have 
clearly  defined  roles  and  responsibilities  and  separate  budget  authority, 
program  execution,  and  functional  authority  for  the  enforcement  of 
national  defense  policy  and  objectives.  As  we  have  reported^  throughout 
the  ciM  initiative,  this  environment  has  promoted  stovepipe  systems 
solutions  in  each  component  agency  and  has  made  it  difficult  to 
implement  departmentwide  oversight  or  visibility  over  information 
resources.  This  same  condition  has  contributed  to  the  difficulty  that  has 
limited  the  Department  in  modernizing  business  processes  and 
implementing  corporate  information  systems  across  service  and  agency 
lines.  This  is  most  evident  in  the  perceived  failure  of  the  Corporate 
Information  Management  initiative  (cm),  which  was  intended  to 
reengineer  business  processes  throughout  the  Department.  In  doing  so, 
the  Department  expected  to  save  bilUons  by  having  more  efficient, 
effective  business  processes  running  across  service  and  component  lines. 
However,  these  benefits  have  yet  to  be  widely  achieved  after  8  years  of 


®We  have  been  assessing  Year  2000  efforts  at  DOD  and  have  issued  reports  on  Year  2000  work  at 
individual  DOD  components.  We  also  plan  to  report  on  DOD’s  overall  response  to  the  Year  2000  issue. 

®As  the  Department’s  CIO,  the  ASD  C3I  is  the  Acquisition  Executive  for  DOD’s  major  information 
systems.  In  this  capacity,  the  staff  who  oversee  major  information  systems  acquisition  for  MAISRC 
report  to  the  ASD  C3I. 

^Defense  ADP:  Corporate  Information  Management  Initiative  Faces  Significant  Challenges 
(GAO/IMTEC-91-35,  April  22, 1991);  Defense  ADP:  Corporate  Information  Management  Initiative  Must 
Overcome  Major  Problems  (GAO/IMTEC-92-77  September  14, 1992);  and  Defense  Management 
Stronger  Support  Needed  for  Corporate  Information  Management  Initiative  to  Succeed 
(GAO/AIMD/NSIAD-94-101,  April  12, 1994). 
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effort.  Without  the  Secretary’s  strong  and  continued  support  for 
management  processes  and  controls  designed  to  improve  information 
management  initiatives,  Clinger-Cohen  Act  implementation  could  well 
suffer  similar  results. 
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In  taking  its  initial  steps  to  implement  the  Clinger-Cohen  Act,  Defense  has 
recognized  that  it  needs  a  better  information  technology  investment 
environment.  However,  implementing  the  act  will  not  be  easy  given 
weaknesses  pervading  the  current  decision-making  8ind  oversight 
environment.  Examples  included  the  following: 

•  dod’s  management  and  oversight  processes  over  information  technology 
projects  are  not  effectively  integrated  to  enable  the  Department  to  manage 
from  a  portfolio  perspective.  Considering  investments  as  a  total  package 
of  possible  projects  is  important  because  it  forces  an  agency  to  decide 
which  projects  are  the  most  critical  to  meeting  mission  needs  and  thus 
should  receive  the  most  resources  and  attention.  Life-cycle  management 
controls  that  provide  oversight  on  a  system-by-system  basis  can 
supplement  and  enhance  this  process,  but  they  cannot  be  a  substitute  for 
it. 

•  DOD  still  has  not  established  information  technology  performance 
measures.  These  need  to  be  in  place  before  a  new  investment  process  can 
begin  so  that  senior  managers  can  begin  comparing  results  being  achieved 
against  projected  costs,  benefits,  and  risks. 

•  As  far  as  visibility  over  costs  and  schedule  are  concerned,  dod  has 
fragments  of  a  mechanism  for  collecting  and  maintaining  all  project 
information.  However  this  mechanism — dist — does  not  maintain  cost 
information  and  its  accuracy  and  reliabihty  are  questionable.  Further,  as 
we  have  recently  reported  to  the  Director  of  the  Defense  Information 
Systems  Agency,^  efforts  to  improve  the  dist  have  been  slow-moving. 

•  While  the  Clinger-Cohen  Act  requires  agencies  to  conduct 
post-implementation  reviews,  dod’s  oversight  processes  have  concentrated 
on  projects  prior  to  their  implementation.  Thus,  in  addition  to  focusing  on 
the  pre-implementation  stages  of  system  life-cycles,  dod  will  have  to 
ensure  that  more  attention  is  given  to  whether  implemented  systems  are 
achieving  the  forecasted  benefits  and  continuing  to  meet  mission  needs 
after  they  are  implemented. 

•  Currently,  dod  does  not  have  an  effective  mechanism  in  place  to  ensure 
that  the  cio  has  complete  and  accurate  information  on  DOD-wide 
compliance  with  technical  and  data  standards.  Such  a  mechanism  is 


*We  recently  reported  in  Defense  Computers:  POP's  Inventory  of  Automated  Information  Systems 
Needs  to  Be  Improved  to  Successfully  Address  Year  2000  Problems  (GAO/AIMD-97-112,  August  13, 
1997)  that  Defense  is  planning  to  increase  the  accuracy  of  the  tool  by  developing  a  purging 
methodology  to  validate  the  data  in  DIST.  At  the  end  of  January  1997,  DIST  officials  told  us  that  it 
would  take  90  days  to  determine  the  methodology.  DOD  reported  that  the  methodology  was  finally 
developed  and  implemented  by  the  end  of  August  1997.  DOD  further  reported  that  it  is  now  checking 
DIST  data  on  a  system-by-system  basis  and  is  developing  parametric  and  consistency  checks  for  the 
data  elements  to  ensure  that  the  required  data  elements  are  fully  and  accurately  populated  by  the 
users. 
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necessary  to  enable  the  cio  to  achieve  the  Department’s  goals  for  systems 
interoperability,  effectively  implement  an  integrated  technical  architecture 
for  the  entire  Department,  and  ensure  the  efficient  exchange  of  standard 
data  among  systems. 

Moreover,  our  review  of  the  migration  strategy  suggests  that  a  real  threat 
to  successful  implementation  of  the  Clinger-Cohen  Act  is  the  Department’s 
consistent  lack  of  adherence  to  sound  decision-making  and  oversight 
processes.  For  example,  at  the  oversight  level,  systems  that  clearly  lacked 
key  pieces  of  technical,  programmatic,  and  economic  justification  were 
allowed  to  go  forward.  At  the  decision-making  level,  many  functional  areas 
did  not  even  bother  to  submit  their  systems  or  analyses  that  supported 
their  decisions  to  department-level  oversight  controls.  If  they  were 
effectively  followed,  the  life-cycle  management  controls  would  have 
helped  ensure  that  sound  business  and  development  practices  were 
followed  for  the  migration  systems. 

Perhaps,  the  greatest  challenge  to  successful  operation,  however,  will  be 
operating  in  an  organizational  environment  that  has  resisted 
departmentwide  oversight  and  visibility  over  information  resources.  For 
example,  the  Department’s  ciM  effort  largely  failed  in  meeting  its  original 
goals  of  bringing  widespread  efficiencies  to  its  business  processes.  The 
Department  was  unable  to  create  meaningful  change  across  organizational 
lines  and  management  support  was  insufficient  to  overcome  initial 
resistance  to  new  ways  of  doing  business.  Efforts  to  implement  the 
Clinger-Cohen  Act  will  require  unwavering  top-level  commitment  to 
overcome  both  organizational  resistance  and  to  institute  meaningful 
controls. 


Recommendations 


To  ensure  that  dod’s  continued  investment  in  migration  systems  provides 
measurable  improvements  in  mission-related  and  administrative 
processes,  we  recommend  that  the  Secretary  of  Defense  require  the 
Defense  components  to  rank  development/modemization  migration 
systems  Justifications  and  complete  them  on  an  expedited  basis.  Further, 
the  Secretary  should  require  the  Chief  Information  Officer  to  review  these 
justifications  and  certify  that  they  include  the  following: 

A  business  case  of  operational  alternatives  for  each  functional  area  that 
clearly  demonstrates  that  continued  development  and  deployment  of  the 
migration  system  is  the  best  solution  for  improving  performance  and 
reducing  costs  in  the  functional  area  it  serves,  when  compared  to  other 
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available  alternatives.  The  alternatives  analyzed  should  include 
reengineering  the  functional  area’s  processes  before  making  investments 
in  information  systems  and  using,  when  appropriate,  the  private  sector  to 
perform  major  functions  now  performed  by  government  personnel  and 
information  systems. 

•  An  economic  analysis  showing  a  return  on  investment  or  other 
results-based  benefits  to  the  Department  that  justify  further  investment  in 
the  migration  system. 

•  Current  compliance  with  applicable  Defense  technical  standards  and  uses 
standard  data,  or  a  schedule  and  plan  for  bringing  the  system  into 
compliance  with  these  standards. 

Lastly,  the  Secretary  of  Defense  should  require  the  Chief  Information 
Officer  to  establish  routine  procedures  for  reporting  on  the  status  of 
reviews  of  migration  system  justifications  to  the  Deputy  Secretary  of 
Defense  so  the  information  can  be  used  in  the  Department’s  planning, 
programming,  and  budgeting  system.  Any  exception  to  the 
accomphshment  of  these  reviews  should  be  approved  by  the  Deputy 
Secretary  of  Defense. 

Further,  we  recommend  that  the  Department’s  Chief  Information  Officer 
revise  Defense’s  policies,  practices,  and  procedures  to  institutionalize  the 
management  of  information  systems  and  technology  expenditures  as 
investments  and  ensure  that  these  investments  provide  measurable 
improvements  in  mission  performance.  In  the  course  of  taking  action  on 
these  matters,  we  recommend  that  the  Secretary  direct  that  the  Chief 
Information  Officer,  in  coordination  with  the  Chief  Financial  Officer  and 
other  appropriate  Defense  officials  take  the  following  actions: 

•  Ensure  that  Defense’s  strategic  information  technology  planning  and 
investment  control  policies,  practices,  and  procedures  include 
requirements  that  Principal  Staff  Assistants  document  that  they  and  the 
key  stakeholders  for  their  functional  areas  (including  the  services, 
agencies,  and  major  commands)  have  (1)  conducted  thorough  economic 
and  risk  analyses  of  alternative  operational  approaches  (business  case 
analyses)  for  accomplishing  the  mission  of  the  functional  area, 

(2)  examined  trade-offs  among  the  competing  proposals,  and 

(3)  prioritized  the  alternative  proposals  based  on  mission  impact,  risk,  cind 
return. 

•  Finalize  and  issue  guidance  for  developing  and  using  analyses  of 
alternatives  and  economic  analyses  for  information  system 
decision-making.  Once  this  guidance  is  issued,  the  cio  should  require  that 
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all  Defense  program  managers  and  Defense  managers,  as  appropriate,  be 
trained  in  using  the  guidance.  Also,  the  cio  should  ensure  that  the  guidance 
defines  a  standard  return  on  investment  definition  for  Defense  information 
systems  and  require  that  this  definition  be  used  to  calculate  all  returns  on 
investment  for  information  systems  efforts.  Additionally,  the  cio  should 
require  that  all  major  migration  and  other  information  systems  under 
direct  review  by  the  Major  Automated  Information  System  Review  Council 
(maisrc),  and  those  delegated  by  maisrc  to  other  oversight  organizations 
for  review,  have  their  alternatives  analyses  and  economic  analyses 
independently  verified  by  Defense’s  Program  Analysis  and  Evaluation 
(pa&e)  office,  or  another  qualified  independent  review  organization,  in 
accordance  with  this  guidance  before  major  investments  are  authorized 
for  system  development/modemization. 

•  Require  post-implementation  reviews  of  migration  and  other  information 
systems  and  ensure  that  these  reviews  are  designed  to  compare  actual 
systems’  costs,  benefits,  risks,  and  returns  against  the  original  baseline 
estimates/projections  and  determine  the  causes  of  any  differences 
between  planned  and  actual  results. 

•  Expedite  the  definition,  coordination,  testing,  and  implementation  of 
information  management  performance  measures  in  the  Department  and 
establish  milestones  for  evaluating  progress  in  implementing  these 
performance  measures. 

•  Modify  the  dist  system  or  acquire/develop  a  new  Defense-wide 
management  information  system  or  systems  for  tracking  and  reporting  key 
schedule,  progress,  and  performance  information  on  migration  systems 
and  other  Defense  information  systems  and  ensure  the  system  or  systems 
contain  complete,  current,  and  accurate 

•  schedule  data  necessary  to  track  the  progress  of  each  migration 
system’s  development/deployment  and  each  legacy  system’s 
termination; 

•  budgeted  and  actual  cost  data  on  each  system  for  which  the  Department 
maintains  such  data  (an  alternative  to  putting  budget  and  cost  data  in 
BIST  is  to  establish  the  capability  to  directly  interface  with  other  systems 
in  the  Under  Secretary  of  Defense  (Comptroller’s)  Office  or  other 
Defense  organizations  containing  systems’  budget  and/or  cost  data); 

•  data  necessary  to  track  the  progress  of  each  migration  system  in 
complying  with  applicable  Defense  technical  and  data  standards, 
including  whether  each  system  has  been  independently  certified  to  be  in 
compliance  with  applicable  technical  and  data  standards; 

•  data  for  tracking  progress  in  accomplishing  the  mission-based 
performance  goals  and  information  management  performance  goals  for 
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the  functional  areas  supported  by  each  system  once  these  data  have 
been  identified; 

•  information  determined  to  be  needed  for  oversight  by  the  Defense 
Acquisition  Board  (dab),  maisrc,  and  pa&e;  and 

•  other  information  determined  to  be  needed  for  management  and 
oversight  by  the  Defense  cio,  the  cio  Council,  other  Defense  senior 
managers,  and  the  Congress. 

•  Develop  and  implement  management  controls  and  a  quality  assurance 
program  to  ensure  the  dist  data’s  accuracy  and  completeness  since  these 
data  are  used  to  track  and  report  to  senior  Defense  managers  and  the 
Congress  on  the  overall  status  of  the  migration  initiative  and  on  other 
Defense  information  management  initiatives. 


Agency  Comments 
and  Our  Evaluation 


The  Department  of  Defense  provided  written  comments  on  a  draft  of  this 
report,  which  are  reprinted  in  appendix  I.  The  Acting  Assistant  Secretary 
of  Defense  for  Command,  Control,  Communications  and  Intelligence 
concurred  with  five  of  our  recommendations  and  partially  concurred  with 
two  recommendations.  Defense  did  not  concur  with  the  remaining  five 
recormnendations  and  expressed  concerns  about  certain  aspects  of  our 
report.  Defense’s  concerns  are  highlighted  and  discussed  below.  Appendix 
I  also  provides  detailed  responses  to  dod’s  views  on  our  recommendations 
and  to  other  specific  comments  on  our  findings. 

Generally,  the  Department  concurred  with  our  recommendations  that  the 
Department  revise  or  develop  internal  policies  and  procedures  to  conform 
to  the  Clinger-Cohen  Act  and  to  improve  the  performance  of  information 
systems  management.  Defense  also  noted  that  the  issues  we  reported  are 
in  the  process  of  being  or  will  be  addressed  as  it  implements  the 
Clinger-Cohen  Act,  the  Federal  Acquisition  Streamlining  Act,  the 
Government  Performance  and  Results  Act,  and  the  Paperwork  Reduction 
Act. 

However,  it  did  not  agree  to  a  recommendation  in  our  draft  that  it  limit 
further  investments  to  expenditures  that  meet  mission  critical  needs  until 
the  investments  are  economically,  functionally,  and  technically  justified 
and  such  justifications  are  independently  reviewed.  In  dod’s  view,  limiting 
migration  system  spending  would  adversely  affect  military  readiness, 
system  development,  and  government  contract  obligations  and  increase 
system  obsolescence.  We  understand  dod’s  concerns  regarding  readiness 
and  contract  obligations.  That  is  why  our  recommendation  in  the  draft 
report  recognized  that  critical  needs  still  need  to  be  met.  Our  point  is  that 
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greater  management  attention  needs  to  be  placed  on  the  decision-making 
process  for  approving  and  funding  the  development  and  modernization  of 
migration  systems  to  achieve  intended  benefits  and  minimize  urmecessary 
costs.  In  order  to  clarify  this  position,  we  modified  our  recommendation  to 
focus  on  strengthening  the  controls  for  the  migration  system 
decision-making  process  so  that  these  investments  can  be  better 
considered  in  the  Department’s  budget  process. 

In  addition,  Defense  did  not  concur  with  a  proposal  in  the  draft  report  that 
it  consider  separating  the  cio  and  the  asd  C3i  positions  so  that  the  cio  can 
devote  full  attention  to  departmentwide  information  resource 
management  issues  and  provide  independent  oversight.  It  noted,  however, 
that  cdl  ASD  C3I  functions  are  being  reviewed  by  the  Department’s  Task 
Force  on  Defense  Reform  as  part  of  its  review  of  the  Office  of  the 
Secretary  of  Defense’s  organizational  structure.  We  withdrew  the  proposal 
because  dod  is  now  considering  this  matter.  Nevertheless,  our  concern 
that  the  current  structure  of  the  cio  position  does  not  allow  the  cio  to 
devote  full  attention  to  critical  irm  issues — such  as  computer  secmity,  the 
Year  2000  problem,  and  the  need  to  develop  and  implement  an  integrated 
information  technology  architecture — ^remains  valid. 

Defense  also  stated  that  our  report  overlooked  one  of  the  key  cost 
reduction  aspects  of  the  migration  effort — ^that  is,  modernizing  multiple 
systems  within  each  service  and  Defense  agency  is  more  expensive  than 
simply  modernizing  one  or  a  few  departmentAvide  migration  systems. 

While  these  cost  reductions  may  be  possible.  Defense  has  not  yet 
demonstrated  that  it  has  achieved  such  savings.  Further,  our  review 
determined  that  Defense  does  not  know,  or  track,  cost  reductions  that 
result  from  its  migration  efforts.  Finally,  the  achievements  that  Defense 
refers  to  m  its  comment  letter  are  mostly  the  results  of  process 
reengineering,  not  the  migration  of  automated  systems. 

Defense  also  stated  that  our  report  negatively  portrays  the  role  and 
benefits  of  acquisition  streamlining  within  the  Department.  Defense 
reported  that  it  uses  a  “best-value”  approach  to  modernizing  its  systems 
that  balances  the  functional  and  technical  capabilities  while  still 
considering  cost.  We  disagree  with  Defense’s  characterization  of  our 
report.  We  fully  support  acquisition  streamlining  efforts  in  dod  because 
they  promote  economy,  efficiency,  and  effectiveness  in  the  procurement 
of  property  and  services.  However,  these  three  elements  must  be  carefully 
balanced  in  order  to  achieve  the  results  intended  by  the  Division  D  of  the 
Clinger-Cohen  Act  of  1996  which  covers  federal  acquisition  reform.  Such  a 
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balance  is  particularly  important  to  achieve  prudent  decisions  for  an 
investment  as  substantial  as  the  multibillion  dollar  migration  initiative,  and 
it  is  contemplated  by  the  Clinger-Cohen  Act  of  1996  to  properly  control 
these  investments. 

In  our  view,  for  the  migration  effort,  it  appears  that  expediency  was 
achieved  at  the  expense  of  economy  and  effectiveness.  In  conducting  our 
review,  we  sought  to  examine  whether  the  Department  of  Defense  was 
following  its  own  policy  for  controlling  migration  investments.  When  we 
found  that  these  policies  were  not  executed  as  intended,  we  turned  to  the 
Department’s  acquisition  review  process,  which  should  have 
supplemented  oversight  for  the  major  migration  systems.  However,  even 
here,  we  found  that  dod  for  the  most  part,  was  not  providing  sufficient 
assurance  that  migration  investments  were  worthwhile.  Specifically,  we 
found  that  controls  that  Defense  built  into  this  process  would  not  ensure 
success. 

In  addition,  the  migration  strategy  is  a  high-risk  endeavor  because  it 
requires  Defense  to  carefully  consider  complex  technical  issues  that  are 
compounded  by  the  sheer  size  of  the  Department,  the  number  of  disparate 
systenis,  and  the  prevailing  cultiu-e,  which  promotes  stovepiped  systems 
solutions  and  hampers  departmentwide  oversight  or  visibility  over 
information  resources.  As  such,  the  success  of  the  strategy  hinges  on 
whether  practical  and  soimd  management  practices — such  as  conducting 
economic,  risk,  and  alternative  analyses  and  providing  rigorous  oversight 
over  the  selection  process — are  followed.  As  we  note  in  the  report,  these 
practices  are  required  by  executive  branch  policies;  congressional  reform 
initiatives,  including  the  Clinger-Cohen  Act,  which  took  effect  August  8, 
1996;  and/or  Defense  regulations.  Under  the  Clinger-Cohen  Act,  for 
example,  Defense  is  required  to  implement  a  process  for  selecting 
information  technology  investments  using  specific  criteria  for  comparing 
and  ranking  alternative  information  system  projects.  The  best  practices  on 
which  the  act  is  based  have  shown  that  to  ensure  the  success  of  an 
investment  process,  top  executives  need  to  make  funding  decisions  based 
on  factors  such  as  cost,  risk,  return  on  investment,  and  support  of 
mission-related  outcomes.  Thus,  unless  Defense  promptly  embraces  the 
need  to  develop  and  review  economic,  risk,  and  alternative  analyses,  it 
stands  little  chance  of  successfully  implementing  the  Clinger-Cohen  Act. 

In  its  comments,  Defense  also  expressed  concern  that  the  scope  of  our 
report  was  unnecessarily  broader  than  it  was  when  we  began  our  review. 
We  were  originally  asked  for  information  on  the  status  and  cost  of  the 
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migration  effort.  However,  when  we  foimd  that  Defense  could  not  provide 
accurate  and  reliable  information  on  migration  or  convincingly 
demonstrate  whether  the  strategy  has  been  successful,  om:  congressional 
requester  asked  that  we  review  whether  Defense’s  management  control 
and  oversight  processes  for  migration  were  ensuring  that  the  investments 
are  economically  sound  and  complied  with  technical  and  data  standards. 
We  reviewed  the  rehability  of  the  acquisition  process  controls  over 
migration  investments  because  they  are  the  leading  Defense-recognized 
controls  over  major  investments.  Our  congressional  requester  also  asked 
that  we  focus  on  dod’s  implementation  of  the  Chnger-Cohen  Act  because  it 
offered  a  chance  for  dod  to  bring  meaningful  reform  to  the  management  of 
migration  and  other  information  technology  investments.  Many 
information  technology  investments  for  systems  in  the  Department  are,  in 
fact,  migration  systems. 

Finally,  Defense’s  comments  on  our  report  identify  a  number  of  other 
disagreements  with  our  findings  and  conclusions.  We  reviewed  these 
comments,  incorporated  them  into  our  report  where  appropriate,  and 
followed  up  with  Defense  on  additional  information  it  provided  after 
submitting  its  response  to  us. 
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Note:  GAO  comments 
supplementing  those  in  the 
report  text  appear  at  the 
end  of  this  appendix. 


See  comment  1 . 


See  comment  2. 


ASSISTANT  SECRETARY  OF  DEFENSE 
6000  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-6000 

September  12,  1997 


COMMAND.  CONTROL. 
COMMUNICATIONS.  AND 
INTELLIGENCE 


Mr.  Gene  L.  Dodaro 
Director/ 

Accounting  and  Information 
Management  Division 
U.S.  General  Accounting  Office 
Washington,  DC  20548 

Dear  Mr.  Dodaro; 

This  is  the  Department  of  Defense  (DoD)  response  to  the 
General  Accounting  Office  (GAO)  Draft  Report,  "DEFENSE  IRM:  Poor 
Implementation  of  Management  Controls  Has  Put  Migration  Strategy 
at  Risk,"  dated  August  5,  1997,  (GAO  Code  511355;  OSD  Case  1427). 

The  Department  has  reviewed  the  draft  report  in  detail  and 
we  are  very  concerned  about  its  scope,  inaccuracies  and  some  of 
the  conclusions  and  recommendations.  Several  of  the 
recommendations  which  the  GAO  would  like  the  DoD  to  initiate  are 
already  underway  and  are  currently  being  addressed  as  part  of  the 
DoD  implementation  of  the  Clinger-Cohen  Act  of  1996.  Specific 
responses  to  the  GAO  recommendations  are  provided  in  Enclosure  1. 

Our  review  of  the  draft  report  reveals  many  conclusions 
unsupported  by  any  continuous  chain  of  logic  and  analysis  and 
based  on  incorrect  or  out-of-date  information.  At  the  same  time, 
the  GAO  appears  to  have  ignored  published  DoD  policy  and  the 
Department's  documented  results  and  accomplishments. 

In  our  view,  the  draft  report  documents  an  excursion  away 
from  the  task  the  Congress  intended  for  the  GAO  to  undertake  as 
stated  in  the  GAO  memorandum  of  June  12,  1996,  announcing  the 
review.  For  example,  in  our  review  of  the  report,  we  fail  to  see 
how  the  review  of  migration  systems  strategies,  specifically 
focusing  on  cost  and  schedule,  translates  to  the  very  broad  GAO 
recommendations  that  focus  on  how  the  Department  should  be 
organized  and  operated. 

Our  review  of  the  draft  report  reveals  questions  about  the 
accuracy  of  the  data  and  the  balanced  portrayal  of  the  facts ,  We 
have  provided  information  and  documentation  that  would  make  the 
report  accurately  portray  the  actual  circumstances,  however,  this 
information  has  been  ignored.  For  example,  these  inaccuracies 

o 
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include  the  changing  of  baselines  on  migration  systems  and  the 
reporting  of  cost  data  and  the  review  and  oversight  of  Major 
Automated  Information  Systems  Review  Council  (MAISRC)  systems, 
especially  as  it  relates  to  economic  analysis.  Specific  comments 
are  provided  in  enclosure  2. 

The  description  in  the  report  of  the  relationship  of  the 
migration  systems  effort  to  the  Corporate  Infoirmation  Management 
(CIM)  initiative  is  inaccurate.  The  GAO  indicates  in  the  report 
See  comment  3.  that  the  migration  effort  is  one  of  two  approaches  of  the  CIM 

effort,  along  with  process  reengineering.  This  grossly  mis¬ 
represents  the  principles  of  the  CIM  initiative  that  involved 
several  aspects:  (1)  corporate  policy /planning,  (2)  process  & 
data  modeling,  (3)  process  improvement,  (4)  performance 
measurement,  (5)  standard  information  systems  and  (6)  computing 
and  communications  infrastructure.  All  these  initiatives  are 
extremely  important  to  effectively  fielding  information  systems 
to  support  the  mission.  To  date  we  have  over  160  major 
improvement  projects  documented.  We  have  an  additional  37 
projects  underway  that  increase  readiness,  reduce  cycle  time  and 
cost,  and  respond  to  the  needs  of  the  warfighter  in  this 
technological  age.  These  projects  range  across  the  whole 
Department  as  the  following  examples  illustrate:  US  Atlantic 
Command's  Information  Intranet,  Civilian  Personnel  Reengineering, 
Defense  Investigative  Service,  Medical  Logistics,  Management  of 
Consumables,  Telemedicine,  Strategic  Warfare  Planning,  and  DoD 
Travel  Reengineering. 

Whether  there  was  a  CIM  initiative  or  not,  the  migration 
effort  should  have  been  initiated  to  achieve  its  primary 
Se©  COmmGnt  4.  objective  to  eliminate  duplication  and  reduce  costs.  The  GAO 

cites  a  figure  of  $18  billion  for  migration  systems  spending 
which  is  misleading  and  overly  simplistic.  Independent  of  the 
migration  effort,  information  systems  must  be  modernized, 
integrated  with  other  systems  and  new  capabilities  added.  The 
concentration  of  the  GAO  on  purely  cost  issues  is  inappropriate. 
Each  migration  selection  must  be  based  upon  a  balanced  assessment 
of  functional  and  technical  capabilities  while  still  considering 
cost.  This  approach  is  preferred  in  order  to  ensure  proper 
consideration  of  the  criticality  of  the  investment  and  to  meet 
the  primary  objective  of  supporting  the  warfighter. 

Additionally,  the  GAO  has  ignored  one  of  the  key  cost  reduction 
aspects  of  the  migration  effort.  The  obvious  benefit  of  reducing 
duplication  is  a  reduction  in  the  operational  cost  of  replaced 
legacy  systems,  which  the  GAO  recognizes  and  comments  on. 

Another  significant  benefit  is  the  cost  avoidance  of  modernizing 
multiple  systems  within  each  Service  and  Defense  Agency  which  is 
more  expensive  than  simply  modernizing  one  or  a  few  DoD-wide 
migration  system (s) .  This  cost  avoidance  frees  up  funds  to 
modernize  other  priority  initiatives. 
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Acquisition  streamlining  is  vilified  in  the  draft  report. 

See  comment  5  process-driven  mentality  of  the  report  ignores  the  reality  of 

acquisition  streamlining  initiatives.  The  report  attempts  to 
apply  methodology  in  the  DoD  5000  series  regulations  without  an 
understanding  of  acquisition  streamlining  initiatives.  The  one 
size-f its-all  approaches  tried  in  the  past,  before  the 
acquisition  streamlining  initiative,  did  not  work  as  well. 
Furthermore,  the  GAO  report  faults  the  Department's  acquisition 
oversight  for  many  issues  that  are  not  acquisition  related.  The 
Department  recognizes  the  need  to  expand  its  oversight  to  address 
the  needs  of  the  Chief  Information  Officer  as  well  as  the  Defense 
Acquisition  Executive,  but  it  is  naive  for  the  GAO  to  assume  that 
the  CIO  will  rely  solely  on  the  current  acquisition  oversight 
process,  developed  prior  to  Clinger-Cohen,  to  oversee  IT 
investments . 

The  statements  that  MAISRC  and  Defense  Acquisition  Board 
(DAB)  Programs  have  not  been  reviewed  for  compliance  with 
See  comment  6.  applicable  DoD  technical  and  data  standards  are  not  accurate. 

All  programs  on  the  Major  Automated  Information  Systems  list  are 
or  will  be  Defense  Information  Infrastructure/Common  Operating 
Environment  compliant.  The  Defense  Information  Systems  Agency 
supports  the  MAISRC  as  a  member  of  Integrated  Product  Teams  by 
validating  all  data  elements  and  technical  standards.  These 
reviews  are  based  on  the  standards  and  policies  that  exist  at  the 
time  of  the  review. 

The  issues  surfaced  in  this  report  have  been  or  will  be 
addressed  with  our  implementation  of  the  Clinger-Cohen  Act, 
Federal  Acquisition  Streamlining  Act,  Government  Performance  and 
Results  Act  and  Paperwork  Reduction  Act.  The  DoD  is  in  the 
process  of  formally  incorporating  the  requirements  of  these  Acts 
into  our  regulatory  guidance  and  oversight  processes. 
Implementation  of  the  requirements  of  these  Acts  will  ensure 
improved  fielding  of  information  technology  to  support  the 
warfighter . 

We  appreciate  the  opportunity  to  comment  on  the  draft 
report . 

Sincerely, 


- /  AntJ&ny  M.  Valletta 

(^ting) 

Enclosures 
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Now  on  p.  51 . 


See  comment  7. 


Now  on  pp.  51-52. 


DoD  Response  to 

GAO  DRAFT  REPORT  -  DATED  AUGUST  5,  1997 
(GAO  CODE  511355)  OSD  CASE  1427 

"DEFENSE  IRH:  POOR  IMPLEMENTATION  OF  MANAGEMENT  CONTROLS  HAS  PUT 
MIGRATION  STRATEGY  AT  RISK  " 

RECOMMENDATIONS 

o  RECOMMENDATION  Ig  The  GAO  recommended  that  the  Secretary  of 
"Defense  limit  further  expenditures  in  the  development/ 
modernization  of  migration  systems  to  those  expenditures  that 
meet  the  critical  needs  of  the  Department.  (p.  71 /GAO  Draft 
Report) 

POD  RESPONSE  TO  THE  DRAFT  REPORT;  Nonconcur.  Neither  the 
findings  or  conclusions  presented  in  the  GAO  report  warrant 
limiting  capital  investments.  We  intend  to  fully  comply  with 
the  spirit  and  intent  of  the  Clinger-Cohen  Act  regarding 
investment  in  information  technology  to  include  developing  a 
capital  planning  and  investment  process  revolving  around  the 
DoD  Planning,  Programming  and  Budgeting  System  (PPBS) .  That 
process  will  include  the  provisions  of  the  Act  regarding  the 
criticality  of  information  technology  investments  to  the 
Department.  As  the  GAO  acknowledged  in  the  report,  the  DoD 
has  taken  a  positive  step  to  turn  around  its  information 
technology  management  process  as  part  of  its  implementation  of 
the  Clinger-Cohen  Act. 

As  previously  indicated  in  response  to  other  GAO  Audit 
Reports,  the  DoD  uses  a  best-value  approach  that  balances 
functional  and  technical  capabilities  while  still  considering 
cost.  This  approach  is  preferred  to  ensure  proper 
consideration  of  the  criticality  of  the  investment  and  to  meet 
the  primary  objective  of  supporting  the  warfighter.  Moreover, 
limiting  expenditures  will  adversely  impact  force  readiness, 
system  development  and  government  contract  obligations. 
Finally,  limiting  expenditures  in  development  and 
modernization  is  counter  to  improving  the  information 
technology  support  to  the  warfighter  and  will  lead  to 
increased  obsolescence  of  our  systems. 

o  RECOMMENDATION  2:  The  GAO  recommended  that  limitations 
on  expenditures  should  continue  until  the  Department's 
Chief  Information  Officer  (CIO)  certifies  that  the 
justifications  —  (1)  a  business  case  of  operational 
alternatives  for  each  functional  area,  (2)  an  economic 
analysis  showing  a  return  on  investment,  and  (3)  current 
compliance  with  applicable  Defense  technical  standards  and 
uses  standard  data- -were  prepared,  independently  verified,  and 
available  for  review  for  each  migration  system. 

(p.  72 /GAO  Draft  Report) 

Elnclosure  1  to  Hemo--GAO 
Draft  Report — OSD  Caae  1427 
Page  1  of  7 
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See  comment  7. 

DOD  RESPONSE  TO  THE  DRAFT  REPORT:  Nonconcur.  See  DoD 
response  to  RECOMMENDATION  1  on  the  implementation  of  the 

Clinger-Cohen  Act,  and  RECOMMENDATIONS  5  and  7,  below. 

Now  on  p.  52. 

o 

RECCniMENDATXON  3:  The  GAO  recommended  that  any  exceptions  to 
these  justifications  for  migration  system  investments  should 
be  documented  and  only  the  Deputy  Secretary  or  the 

Department's  Chief  Information  Officer  should  be  authorized  to 
grant  exceptions.  {p.  72 /GAO  Draft  Report) 

See  comment  7. 

DOD  RESPONSE  TO  THE  DRAFT  REPORT:  Nonconcur.  See  DoD 
response  to  RECOMMENDATION  1  on  the  implementation  of  the 

Clinger-Cohen  Act. 

Now  on  p.  52. 

o 

RECOMMENDATION  4:  The  GAO  recommended  that  the  Chief 

Information  Officer  revise  Defense's  policies,  practices,  and 
procedures  to  institutionalize  the  management  of  information 
systems  and  technology  expenditures  as  investments  and  ensure 
that  these  investments  provide  measurable  improvements  in 
mission  performance.  (p.  72 /GAO  Draft  Report) 

DOD  RESPONSE  TO  THE  DRAFT  REPORT:  Concur.  DoD  policies, 
practices  and  procedures  are  being  revised  to  satisfy  the 
provisions  of  the  Clinger-Cohen  Act.  The  Secretary  of  Defense 
has  published  a  policy  memorandum  on  implementation  of  the 

Clinger-Cohen  Act.  The  Department  is  reviewing,  and  will 
ultimately  revise,  a  variety  of  DoD  Directives  and 

Instructions  to  reflect  information  technology  investment 
policy.  Once  these  policies  are  coordinated,  they  will  be 
formally  issued. 

i 

Now  on  p.  52. 

o 

RECOMMENDATION  5:  The  GAO  recommended  that  the  Secretary 
direct  that  the  Chief  Information  Officer,  in  coordination 
with  the  Chief  Financial  Officer  (CFO)  and  other  appropriate 

Defense  officials  ensure  that  Defense's  strategic  information 
technology  planning  and  investment  control  policies, 
practices,  and  procedures  include  requirements  that  each 

Principal  Staff  Assistant  document  that  they  and  the  key  stake 
holders  for  their  functional  area  (including  the  Services, 

Agencies,  and  Major  Commands)  have  conducted  thorough  economic 
and  risk  analyses  of  alternative  operational  approaches 
(business  case  analyses)  for  accomplishing  the  mission  of  the 
functional  area,  examined  trade-offs  among  the  competing 
proposals,  and  prioritized  the  alternative  proposals  based  on 
mission  impact,  risk,  and  return.  (p.  72-73 /GAO  Draft  Report) 

See  comment  8. 

DOD  RESPONSE  TO  THE  DRAFT  REPORT:  Partially  concur. 

Policies,  practices  and  procedures  relating  to  information 
technology  planning  and  capital  investment  are  being  revised 
to  satisfy  the  provisions  of  the  Clinger-Cohen  Act,  The 

Secretary  of  Defense  has  published  a  policy  memorandum  on 
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implementation  of  the  Clinger-Cohen  Act.  The  Department  is 
reviewing,  and  will  ultimately  revise,  a  variety  of  DoD 
Directives  and  Instructions  to  reflect  information  technology 
investment  policy.  Once  these  policies  are  coordinated,  they 
will  be  formally  issued. 

o  RKCOMMENDATION  6;  The  GAO  recommended  that  the  Secretary 
direct  that  the  Chief  Information  Officer,  in  coordination 
with  the  Chief  Financial  Officer  and  other  appropriate  Defense 
officials  finalize  and  issue  guidance  for  developing  and  using 
analyses  of  alternatives  and  economic  analyses  for  information 
system  decisionmaking.  Once  this  guidance  is  issued,  the  CIO 
should  require  that  all  Defense  program  managers  and  Defense 
managers,  as  appropriate,  be  trained  in  using  the  guidance. 
Also,  the  CIO  should  ensure  the  guidance  defines  a  standard 
return  on  investment  definition  for  Defense  information 
systems  and  require  this  definition  be  used  to  calculate  all 
returns  on  investment  for  information  systems  efforts.  (p. 

Now  on  p.  52.  73/GAO  Draft  Report) 

DOD  RESPONSE  TOTHB  DRAFT  REPORT;  Partially  concur.  DoDI 
7041 . 3 ,  "Economic  Analysis  for  Decisionmaking, "  dated 
November  7,  1995,  indicates  the  minimum  standards  that  an 

See  comment  9.  economic  analysis  must  meet  in  order  to  be  considered 

reasonably  adequate.  Within  the  analytic  framework  provided 
therein.  Department  analysts  may  apply  those  methods  of 
analysis,  including  various  tools,  format  displays,  estimating 
relationships  and  models,  that  are  most  appropriate  to  the 
acquisition  program  under  analysis.  The  depth  and  accuracy  of 
each  effort  is  determined  by  the  size,  impact  and  risk 
associated  with  the  decision  that  the  analysis  is  required  to 
support.  In  addition  to  DoDI  7041.3,  PA&E  has  developed  and 
published  an  unofficial  Automated  Information  System  (AIS) 
Economic  Analysis  (EA)  Guide  that  is  designed  to  assist  in  the 
development  of  a  standard  approach  to  Information  Technology 
(IT)  EA.  This  Guide,  however,  recommends  that  the  standard 
methods  and  formats  provided  be  specifically  tailored  for  each 
application,  and  is  in  keeping  with  acquisition  streamlining. 
This  approach  has  allowed  flexibility  and  substantial  economy 
in  the  IT  EA  process .  It  has  been  adequate  to  support  the 
present  Major  Automated  Information  Systems  Review  Council 
(MAISRC)  system  wherein  each  Major  AIS  is  reviewed 
individually. 

We  recognize  that  greater  standardization  may  be  required 
in  the  EA  process  in  order  to  successfully  implement  a 
portfolio  oversight  process.  A  portfolio  approach  will 
require  that  different  programs  be  evaluated  against  each 
other,  hence,  standard  methods  of  measurement,  analytic 
techniques,  indicators  and  measure  of  merit  will  be  required. 
In  conjunction  with  the  implementation  of  a  portfolio 
approach,  therefore,  the  Department  will  officially  publish 
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appropriate  documents  to  provide  minimum  standards  and 
guidance  for  the  EA  process,  yet,  hopefully,  maintain  a 
practicable  level  of  flexibility  required  in  order  to  address 
program  differences  and  new  analytic  techniques. 

o  RECOMMENDATION  7 :  The  GAO  recommended  that  the  CIO  should 

r equ ire  tha t  all  major  migration  and  other  information  systems 
under  direct  review  by  Major  Automated  Information  System 
Review  Council  (MAI SRC)  and  those  delegated  by  MAI SRC  to  other 
oversight  organizations  for  review,  have  their  alternatives 
analyses  and  economic  analyses  independently  verified  by 
Defense's  Program  Analysis  and  Evaluation  (PA&E)  office  in 
accordance  with  this  [analysis  of  alternatives  and  economic 
analysis]  guidance  before  major  investments  are  authorized  for 
Nowon  53  system  development/modernization.  (p.  73/GAO  Draft  Report) 

POD  RESPONSE  TO  THE  DRAFT  REPORT:  Nonconcur.  The  DoD  5000.2- 
R  provides  guidance  in  section  2.4.  for  the  Analysis  of 
Alternatives  (AoAs)  and  in  section  3.5  for  Cost  Analysis. 

Nonconcur  with  recommendation  that  DoD  Program  Analysis 
and  Evaluation  Directorate  (PA&E)  perform  an  independent 
See  COrriment  10.  verification  of  analyses  of  alternatives.  The  Department 

requires  that  an  AoA  be  prepared  by  the  Principal  Staff 
Assistant  at  Milestone  0.  DoD  policy  also  requires  the 
Principal  Staff  Assistant  to  coordinate  with  PA&E  and  others 
early  in  the  development  of  the  AoAs .  This  approach  is 
consistent  with  the  Integrated  Product  Team  (IPT)  approach  the 
Department  has  adopted  for  the  conduct  of  acquisition  and 
oversight.  The  DoD  directive  does  not  support  the  GAO 
recommendation  that  the  PA&E  perform  an  independent 
verification  of  the  analysis.  Department  policy  also  requires 
acquisition  program  managers  to  prepare  an  Economic  Analysis 
at  program  initiation.  The  policy  requires  the  Principal 
Staff  Assistant  or  Component  to  ensure  that  a  Component  cost 
analysis  is  conducted  at  program  initiation.  That  analysis 
and  PA&E's  participation  in  all  acquisition  program  Cost  IPTs 
accomplish  the  independent  review  of  economic  analyses  GAO 
proposes.  PA&E  also  reviews  and  coordinates  on  all  MAISRC 
Acquisition  Decision  Memoranda  (ADM) . 

Further,  only  those  systems  reviewed  directly  by  the  OSD 
MAISRC  require  review  by  PA&E.  The  responsibility  for 
ensuring  that  analysis  of  a  delegated  program  has  been 
completed  and  is  valid  is  delegated  to  the  Component.  The  DoD 
Component  head,  or  delegated  authority,  has  the  responsibility 
for  determining  the  independent  activity  responsible  for 
preparing  an  analysis.  To  require  otherwise  would  defeat  the 
goal  of  delegation. 

o  RECOMMENDATION  8;  The  GAO  recommended  that  the  Secretary 
direct  that  the  Chief  Information  Officer,  in  coordination 

Enclosure  1  to  Memo--GAO 
Draft  Report “~OSD  Case  1427 
Page  4  of  7 


Page  64 


GAO/AIMD-98-5  Defense  Migration  Strategy 


Appendix  I 

Comments  From  the  Department  of  Defense 


Now  on  p.  53. 


Now  on  p.  53. 


Now  on  p.  53. 


with  the  Chief  Financial  Officer  and  other  appropriate  Defense 
officials  require  post -implementation  reviews  of  migration  and 
other  information  systems  and  ensure  these  reviews  are^ 
designed  to  compare  actual  systems'  costs,  benefits,  risks, 
and  returns  against  the  original  baseline 
estimates /projections  and  determine  the  causes  of  any 
differences  between  planned  and  actual  results.  (p.  73 /GAO 
Draft  Report) 

POD  RESPONSE  TO  THE  DRAFT  REPORT;  Concur.  The  acquisition 
community  is  already  taking  steps  to  implement  this 
recommendation  for  information  systems .  Recent  MAISRC 
Milestone  III  ADMs  have  included  the  requirement  for  post- 
deployment  performance  evaluations  and  will  continue  to 
require  these,  as  appropriate. 

o  RECOMMENDATION  9:  The  GAO  recommended  that  the  Secretary 
direct  that  the  Chief  Information  Officer,  in  coordination 
with  the  Chief  Financial  Officer  and  other  appropriate  Defense 
officials  expedite  the  definition,  coordination,  testing,  and 
implementation  of  information  management  performance  measures 
in  the  Department  and  establish  milestones  for  evaluating  the 
progress  of  implementing  these  performance  measures.  (p. 

74 /GAO  Draft  Report) 

POD  RESPONSE  TO  THE  DRAFT  REPORT;  Concur.  The  Department 
published  an  investment  and  measurement  guide  and  revised 
internal  regulations  to  incorporate  GPRA  and  Clinger-Cohen 
performance  measurement  requirements.  In  March  1997,  the  DoD 
CIO  approved  the  ITM  Strategic  Plan  and  directed  Components  to 
develop  ITM  Strategic  Plans  and  investment  criteria  to  be 
implemented  during  the  next  PPBS  cycle  starting  in  October 
1997 .  Component  ITM  Strategic  Plans  have  been  received 
covering  over  95%  of  the  DoD  Exhibit  IT-43  resources.  In 
compliance  with  the  Clinger-Cohen  Act  and  0MB  guidance,  the 
plans  create  specific  strategic  links  to  DoD  missions  and 
define  performance  indicators  to  measure  progress.  The  DoD 
CIO  has  directed  the  development  of  specific  performance 
indicators  and  a  performance  plan  to  focus  CIO  attention  on 
achieving  desired  mission  and  IT  support  outcomes  described  in 
the  DoD  ITM  Strategic  Plan  and  influence  future  Department 
plans  and  programs  at  all  levels. 

o  RECOMMENDATION  10 t  The  GAO  recommended  that  the  Secretary 
direct  that  the  Chief  Information  Officer,  in  coordination 
with  the  Chief  Financial  Officer  and  other  appropriate  Defense 
officials  modify  the  Defense  Integrated  Support  Tools  (DIST) 
system  or  acquire/develop  a  new  Defense-wide  management 
information  system  or  systems  for  tracking  and  reporting  key 
schedule,  progress,  and  performance  information  on  migration 
systems  and  other  Defense  information  systems.  (p.  75 /GAO 
Draft  Report) 
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Now  on  p.  54. 


POD  RESPONSE  TO  THE  DRAFT  REPORT;  Concur.  The  DIST  is  the 
Department's  official  automated  repository  and  backbone 
management  tool  for  DoD's  inventory  of  systems.  It  contains 
information  on  hardware  platforms,  operating  systems, 
application  languages,  communications,  and  data  interfaces. 

Since  the  fall  of  1996,  the  Department  has  taken  major  actions 
to  enhance  the  DIST  with  emphasis  on  three  areas:  ease  of  use, 
user  access,  and  collection  of  information  needed  to  address  the 
Year  2000  problem.  Actions  are  now  underway  to  re-engineer  the 
migration  tracking  tool  and  add  additional  fields  to  the 
database  required  to  satisfy  the  needs  of  the  DOD  CIOs.  The 
Defense  Information  Systems  Agency  (DISA)  and  ASD{C3I)  are 
working  closely  with  the  CIO  Council  and  DIST  User's  Group  to 
ensure  all  requirements  are  documented  and  prioritized  for 
implementation.  An  initial  release  of  the  new  migration 
tracking  tool  is  scheduled  for  delivery  on  October  30,  1997- 

DISA  and  ASD(C3I)  are  working  with  the  staff  of  the  Under 
Secretary  of  Defense  (Comptroller)  to  develop  links  between  the 
planning  and  budgeting  systems  and  the  DIST.  For  the  current 
Program  Objective  Memoranda  (POM)  Planning  Cycle,  the  DOD 
Components  are  required  to  use  the  DIST  Identification  number 
when  reporting  system  costs. 

o  RECOMMENDATION  11;  The  GAO  recommended  that  the  Secretary  of 
Defense  direct  the  Chief  Information  Officer,  in  coordination 
with  the  Chief  Financial  Officer  and  other  appropriate  Defense 
officials  to  develop  and  implement  management  controls  and  a 
quality  assurance  program  to  ensure  the  DIST  data's  accuracy 
and  completeness  since  these  data  are  used  to  track  and  report 
to  senior  Defense  managers  and  the  Congress  on  the  overall 
status  of  the  migration  initiative  and  on  other  Defense 
information  management  initiatives.  (p.  75/GAO  Draft  Report) 

DOD  RESPONSE  TO  THE  DRAFT  REPORT;  Concur.  Beginning  in  June 
1996 ,  the  Defense  Information  Systems  Agency  (DISA) ,  the  DoD 
organization  that  operates  and  maintains  the  DIST,  has 
instituted  a  data  base  validation  and  data  quality  program  to 
ensure  that  the  DIST  contains  accurate  and  complete  data. 

Data  are  being  analyzed  and  suspect  data  (duplicative, 
inactive,  and  incomplete  entries)  are  being  reviewed  with  the 
Authorized  Editors.  The  Department  also  plans  to  perform 
statistical  sampling  of  DIST  data  to  validate  accuracy,  and 
will  rely  on  the  DODIG,  during  its  Year  2000  audits,  to 
validate  DIST  data.  The  Department  will  closely  monitor  this 
program  to  ensure  that  the  data  quality  of  the  DIST  is  at  the 
highest  level  as  required  for  reports  to  senior  Defense 
managers  and  the  Congress. 

o  RECOMMENDATION  12;  The  GAO  recommended  that  the  Secretary 

consider  separating  the  CIO  and  Assistant  Secretary  of  Defense 
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See  comment  1 1 . 


for  Command,  Control,  Communications  and  Intelligence  (ASD 
C3I)  positions  so  that  the  CIO  can  devote  full  attention  to 
department -wide  information  resource  management  (IRM)  issues 
and  provide  independent  oversight.  (p.  75 /GAO  Draft  Report) 

POD  RESPONSE  TO  THE  DRAFT  REPORT;  Nonconcur.  However,  all 
ASD(C3I)  functions  are  being  reviewed  by  the  Secretary  of 
Defense  in  his  study  of  OSD  and  the  Defense  Agencies. 
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Now  on  p.  33. 


Now  on  p.  32. 


OFFrCE  OF  THE  SECRETARY  OF  DEFENSE 
1800  DEFENSE  PENTAGON 
WASHINGTON.  DC  20301-1800 


August  28, 1997 

MEMORANDUM  FOR  OASD(ai)  ATTN.  MR.  KENNETH  GLASSER 
SUBJECT:  Comments  on  GAO  Repon  ‘T^efcnse  KMT  GAO/AIMD-97-105,  Aug.  1997 

In  geaexaZ,  we  concur  with  the  comments  and  xecommendadons  of  the  subject 
We  support,  fiiUy,  the  lecommendarion  to  implement  a  ponfblio  FT  oversight  process. 
Thiec  issues,  however,  merit  funher  commenL 

The  first  concerns  the  report’s  list  of  fourteen  Major  AIS  systems  for  which  PA&E 
analysts  had  indicated  no  knowledge  of  an  Economic  Analysis  (p.41).  Seven  of  these 
systems  were  delegated  to  DoD  Component  oversi^it,  hence,  PA&E  analysts  would  sot 
noBcmally  review  their  supporting  EAs.  Five  of  the  systems  were  characterized  as  at  or 
still  prior  to  Milestone  0  review,  hence,  detailed  £A  would  not  be  available  for  these 
programs.  Respectively,  the  lack  of  PA&E  review  of  the  EAs  for  these  programs  despite 
substantial  investment,  appear  to  indicaie  that  the  MAISRC  may  prematurely  delegate 
Major  AIS  programs  and  that  it  does  not  continue  to  police  them  subsequent  to 
delegation.  Further,  the  lack  of  EAs  for  programs  that  have  expended  substantial 
resources  and  still  have  not  had  MHestone  0  approval,  may  indicate  that  the  MAISRC 
does  not  have  adequate  means  to  determine  programs  meet  MAISRC  thresholds  until 

after  subscandal  investment  has  occurred.  Both  of  these  problems  stem  from  MAISRC 
cognizance  and  scheduling  rather  than  fiom  £A  problems. 

The  second  issue  regards  that  statement  (p.  40)  that  the  Department  has  not  yei 
established  a  set  of  minimum  standards  that  an  £A  must  meet  to  be  considered  valid. 
Dqtaitmeat  of  Defense  Instruction  70413,  ‘^Economic  Analysis  For  Decisionmaking", 
(November  7, 1995),  indicates  the  minimum  standards  that  an  Economic  Analysis  must 
meet  in  order  to  be  considered  reasonably  adequate.  \^ihin  the  analytic  fiamewock 
provided  therein.  Department  analysts  may  apply  those  methods  of  analysis,  including 
various  tools,  foncat  displays,  esdinaxing  rel^ooships  and  models,  that  are  most 
j^jpropriaie  to  die  acqmsitioD  program  under  analysis.  The  dq)ih  and  accuracy  of  each 
effect  is  detenmned  by  the  size,  impact  and  risk  associated  with  the  decision  that  the 
analysis  is  requiied  to  supposrt.  In  addition  to  D6DI 70413,  PA&E  has  developed  and 
pubhriied  an  unofficial  AIS  HA  Guide  that  is  designed  to  assist  in  the  development  of  a 
standard  approach  to  IT  EA.  This  Guide,  however,  recommends  that  the  standard 
methods  and  formats  provided  be  specifically  tailored  for  each  application,  and  is  in 
keeping  with  acquisition  sueamlinmg.  This  approach  has  allowed  flexibility  and 
substantial  economy  in  the  IT  EA  process.  It  has  been  adequate  to  support  the  present 
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MAISRC  systfizn  wherein  each  Major  AIS  is  reviewed  individually.  Wc  recognize, 
however,  that  greater  standardizanon  may  be  required  in  the  EA  process  in  order  to 
successfully  implement  a  portfolio  oversight  process.  A  portfolio  approach  will  require 
that  different  programs  be  equated  against  each  other,  hence,  standard  methods  of 
measurement,  analyde  tedmiques,  indicators,  and  measures  of  merit  will  be  xequiied.  In 
conjunction  with  tte  implementadon  of  a  portfolio  approach,  therefore,  the  Department 
will  officially  publish  appropriate  documents  to  provide  minimum  standards  and  guidance 
for  EA  process,  yet,  hopefully,  maintain  the  highest  practicable  level  of  flexibility 
required  in  Older  to  address  program  differences  and  new  analytic  techniques. 

The  thiid  issw  concerns  the  disposition  of  the  fourteen  Major  AIS  programs 
Now  on  p.  33 .  (p*  41)  as  not  having  submitted  £A*s  for  independent  review  by  the  MAISRC. 

Acquisition  oversight  and  conesponding  independent  EA  reviews  for  seven  of 
systems  (AFMSS,  Army  GCCS,  COPS,  NTCS-Afloar,  OSS,  TSC  and  TOPS)  were 
delected  to  the  DoD  Components.  The  MAISRC  and  PA&E  will  review  these 
only  if  special  conditions  such  as  a  baseline  breach  occur.  Three  of  the  programs  (ADS  as 
pan  of  CHC^  n,  DPS  andTCAIMS  H)  were  at  the  prc-Milestonc  I  level  whaeat 
comprehensive  EA’s  are  still  being  developed  PA&E  willievicw  these  programs  piiorto 
their  next  milestone  reviews.  Two  of  the  programs  (GCCS  and  DJMPS)  became 
MAISRC  programs  late  in  their  development  cycles.  Partial  independent  EA’s  for  thesg 
programs  have  been  amducted  and  future  investments  wfll  be  independently  reviewed 
^  icprcsenL  Finally,  two  of  the  programs  (DM  and  MM) 

eiqsenenoed  a  hi^  level  of  baseline  flux  due  to  program  funding  xeducti^s  leocmuneoded 
by  PA&E  based  on  independent  EA  assessments.  Both  of  these  programs  have 
subsequendy  been  terminated,^  - 


Ronald  C  wusem 
OD(PA&E)  Analyst 
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The  following  are  gao’s  comments  on  the  Department  of  Defense’s  letter 
dated  September  12, 1997. 


GAO  Comments 


1.  The  scope  of  our  work  was  directed  by  the  Congress.  Defense  is  correct 
that  the  initial  scope  was  expanded,  at  the  direction  of  our  congressional 
requester  because  Defense  was  unable  to  provide  complete  and  accurate 
information  to  answer  the  initial  questions.  These  questions  were  (1)  What 
are  the  reported  costs  for  migration  systems  and  the  legacy  systems 
Defense  expects  to  terminate  for  fiscal  years  1995, 1996,  and  1997? 

(2)  How  many  and  what  type  of  migration  systems  did  Defense  designate, 
how  many  legacy  systems  did  Defense  terminate,  and  how  many  are 
scheduled  for  future  termination?  and  (3)  What  economic  studies  did 
Defense  prepare  and  use  to  justify  its  migration  actions?  Because  Defense 
was  unable  to  provide  satisfactory  answers  to  these  initial  questions,  our 
congressional  requester  directed  us  to  expand  our  work  to  evaluate 
Defense’s  management  controls  over  decision-making  for  selecting  and 
implementing  migration  systems.  When  we  found  that  Defense  was  not 
following  the  Department-level  management  controls  that  had  been 
established  for  overseeing  the  selection  and  implementation  of  migration 
systems,  we  looked  for  other  Department-level  management  controls  over 
migration  systems.  We  found  that  Defense’s  oversight  of  major  systems 
acquisitions  by  maisrc  and  dab  was  one  of  the  few  Department-level 
management  controls  over  migration  systems.  However,  because  these 
management  controls  were  hmited  to  the  major  migration  systems,  we 
looked  for  Defense  actions  that  would  increase  Department-level 
management  control  over  all  migration  systems.  This  led  us  to  examine 
Defense’s  actions  to  implement  the  Clinger-Cohen  Act  of  1996.  Throughout 
our  review,  we  kept  our  congressional  requester  and  our  Defense  point  of 
contact  informed  about  changes  in  the  scope  of  our  work. 

2.  We  reviewed  the  accuracy  of  all  data  in  the  report,  especially  data  that 
Defense  questioned  in  tables,  figures,  and  the  text  regarding  (1)  the 
number  of  migration  systems  selected  by  the  functional  managers  and 
approved  at  the  Department  level,  (2)  the  costs  Defense  reported  to  us  for 
migration  systems,  and  (3)  our  findings  related  to  the  preparation  and 
independent  review  of  economic  analyses  for  major  systems  under  review 
by  MAISRC.  We  contacted  the  appropriate  Defense  officials  and  discussed 
each  instance  in  which  Defense  questioned  the  accuracy  of  data.  We  also 
requested  that  these  officials  provide  us  additional  documentation 
pertaining  to  the  data  in  question.  Based  on  these  discussions  and  our 
review  of  the  additional  documentation,  we  made  changes  where 
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appropriate.  In  the  few  cases  in  which  changes  were  warranted,  we 
changed  the  table,  figure,  or  text. 

3.  We  agree  that  ciM  consists  of  several  aspects,  and  we  stated  this  in  a 
footnote  in  the  report.  However,  our  work  showed  that  as  cm  was 
implemented.  Defense  emphasized  two  ways  of  achieving  process 
improvements  and  addressing  problems  associated  with  its  disparate  and 
stovepiped  information  technology  environment:  (1)  business  process 
reengineering  and  (2)  migration. 

4.  As  we  state  in  our  report,  the  $18  bilUon  dollar  estimate  for  developing, 
maintaining,  and  deploying  migration  systems  for  fiscal  years  1995  through 
2000  is  based  on  cost  information  provided  to  us  by  Defense.  Defense 
provided  essentially  the  same  information  to  the  Congress  in  April  1996. 
We  agree  it  is  incomplete  and  inaccurate,  and  we  stated  so  in  our  report. 

In  addition,  we  agree  that  migration  selections  should  be  based  on  a 
balanced  assessment  of  functional,  technical,  and  cost  factors.  However, 
we  foimd  that  Defense’s  selections  placed  very  little  emphasis  on  costs 
and  related  economic  factors.  Our  work  showed  that  in  Defense,  the 
preparation  of  an  economic  analysis  is  often  a  paper  exercise  to  produce  a 
document  that  decisionmakers  do  not  use  when  deciding  whether  to 
invest  in  an  information  system  development  or  modernization  project. 
Additionally,  we  found  that  in  many  cases.  Defense  placed  little 
importance  on  having  complete  and  accurate  information  on  system 
development/modemization  projects’  estimated  costs  and  benefits  at  the 
time  decisions  were  made  to  invest  in  the  projects. 

Further,  we  agree  with  Defense’s  contention  that  modernizing  multiple 
systems  would  normally  be  expected  to  be  more  expensive  than 
modernizing  a  smaller  number  of  migration  systems.  In  fact,  our  report 
does  not  take  issue  with  the  migration  strategy’s  potential  to  reduce  costs. 
Rather,  it  shows  that  Defense’s  poor  management  controls  over  migration 
systems  may  have  kept  Defense  firom  achieving  expected  cost  reductions 
or,  worse  yet,  resulted  in  wasting  hundreds  of  millions  of  dollars  on  failed 
efforts  such  as  the  materiel  management  migration  initiative.  Also, 
because  Defense  has  not  tracked  the  cost  reductions  and  other 
measurable  improvements  that  may  have  been  achieved  through 
migration.  Defense  cannot  demonstrate  the  extent  that  savings  are  actually 
occurring. 
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5.  We  fully  support  acquisition  streamlining  initiatives  and  encourage 
improvements.  However,  a  balance  between  function,  technology,  and 
cost  is  as  necessary  as  a  balance  between  acquisition  streamlining  and 
effective  management  controls  to  reduce  risk.  We  found  that  Defense’s 
streamlined  approach  to  acquisition  oversight  and  its  revisions  of  its 
acquisition  policy  in  March  1996  resulted  in  inadequate  management 
control  over  major  systems  acquisition  projects.  Specifically,  acquisition 
processes  did  not  fully  ensure  that  economic  analyses  for  migration 
projects  were  prepared  and  reviewed  and  that  systems  complied  with 
technical  and  data  standards.  This  increased  the  risk  that  functional  areas 
would  develop  faulty  cost  estimates  and  risk  assessments  and  pursue 
flawed  efforts  without  rigorous  department-level  oversight. 

While  we  do  not  support  a  one-size-fits-all  approach  to  acquisition 
oversight,  we  believe  that  Defense’s  existing  acquisition  oversight 
processes  do  not  require  the  degree  of  standardization  in  methodology  and 
analytic  techniques  to  ensure  the  preparation  of  the  economic  analyses 
necessary  to  support  a  portfolio  management  approach  for  information 
technology  investments,  as  required  by  the  Clinger-Cohen  Act  of  1996. 
Further,  under  omb’s  Capital  Progranuning  Guide,  which  was  issued  in 
July  1997,  Defense  will  be  required  to  perform  an  economic  analysis,  along 
with  other  extensive  planning  steps,  before  submitting  a  budget  proposal 
for  major  information  technology  projects. 

If  Defense  is  to  effectively  implement  the  Clinger-Cohen  Act  of  1996  and 
integrate  its  requirements  with  those  of  the  Government  Performance  and 
Results  Act,  the  Paperwork  Reduction  Act,  and  the  Chief  Financial 
Officers  Act,  changes  will  be  necessary  not  only  in  Defense’s  acquisition 
oversight  processes  but  also  in  its  other  information  technology 
investment  and  management  control  processes.  Further,  to  effectively 
implement  the  requirements  specified  in  these  laws,  the  Chief  Financial 
Officer,  Defense  Acquisition  Executive,  and  leaders  of  the  services, 
agencies,  and  major  commands  must  work  cooperatively  with  the  cio. 

6.  We  recognize  that  disa  reviewed  system  documentation  for  many  of  the 
major  migration  systems  to  determine  if  it  indicated  that  interoperability 
issues  were  being  addressed  or  were  intended  to  be  addressed  as  the 
system  was  developed.  However,  our  work  showed  that  disa  does  not 
regularly  report  detailed  information  to  maisrc  and  dab  that  would  enable 
these  oversight  organizations  to  ensure  that  each  major  system  is 
adequately  complying  with  applicable  technical  and  data  standards.  For 
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example,  disa  does  not  regularly  report  to  maisrc  and  dab  such  detailed 
information  as 

•  each  system’s  current  compliance  with  applicable  standards — that  is,  each 
system’s  current  status  relative  to  the  eight  levels  that  Defense  has  defined 
for  the  Defense  Information  Infrastructure  Common  Operating 
Environment  and  each  system’s  number  and  percentage  of  data  elements 
that  have  been  approved  as  Defense  data  standards  in  the  Defense  Data 
Dictionary  System; 

•  whether  each  program  office  prepared  sound  strategies,  schedules,  and 
cost  estimates  for  achieving  compliance  with  applicable  standards; 

•  whether  each  system’s  compliance  strategy  and  cost  estimate  were 
approved  by  disa; 

•  each  system’s  current  schedule  and  cost  status  for  achieving  compliance 
compared  against  its  baseline  schedule  and  cost  estimate;  and 

•  whether  each  system  has  been  independently  certified  by  disa’s  Joint 
Interoperability  Testing  Command  to  be  in  compliance  with  the  technical 
and  data  standards. 

MAISRC  analysts  responsible  for  overseeing  the  major  migration  systems 
confirmed  that  they  did  not  know  (1)  many  of  the  systems’  current  status 
regarding  compliance  with  applicable  technical  standards  and  use  of 
Defense  standard  data,  or  (2)  whether  the  program  mamagers  had 
developed  and  were  following  sound  strategies  for  bringing  the  systems 
into  compliance. 

As  noted  in  our  report.  Defense’s  technical  and  data  standards  are 
supposed  to  help  pave  the  way  to  an  interoperable  systems  environment. 
However,  without  complete  and  accurate  data  on  individual  systems’ 
compliance  with  standards,  maisrc  and  dab  or  any  other  dod  manager 
cannot  assure  the  cio  that  Defense’s  major  information  systems  are 
complying  with  the  standards.  Without  this  information,  as  well  as 
standards  compliance  information  from  the  managers  of  Defense’s 
nonmajor  systems.  Defense’s  cio  cannot  gauge  the  Department’s  progress 
toward  achieving  its  goal  of  an  interoperable  systems  environment. 

7.  As  noted  in  the  “Agency  Comments  and  Our  Evaluation”  section  of  our 
report,  we  modified  our  recommendation  to  focus  on  strengthening  the 
controls  for  the  migration  system  decision-making  process  so  that  these 
investments  can  be  better  considered  in  the  Department’s  budget  process. 
Nevertheless,  we  remain  concerned  that  if  Defense  proceeds  with  its 
decisions  to  develop/modemize  major  migration  systems  without  ensuring 
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that  they  are  based  on  sound  investment  decision-making  practices,  it  will 
risk  some  portion  of  these  investments  not  providing  measurable 
improvements  in  mission  performance  for  the  warfighter.  Defense  will 
also  risk  the  systems’  not  operating  as  intended  and  not  being 
interoperable  with  other  systems. 

We  have  reported  on  numerous  occasions  that  poorly  planned  and 
managed  systems  development/modemization  efforts  often  fail  and  must 
be  restarted,  delaying  the  delivery  of  systems  that  can  effectively  meet  the 
warfighter’s  needs.  This  wastes  the  limited  funds  Defense  has  available  for 
providing  information  technology  support  to  the  warfighter  and 
unnecessarily  diverts  funds  from  other  needed  investments.  For  example, 
our  review  of  Defense’s  materiel  management  migration  strategy  showed 
that  Defense  spent  over  $700  million  pursuing  a  substantially  flawed  effort 
that  was  later  abandoned.  We  found  that  Defense’s  planning  and 
decision-making  for  this  strategy  were  not  sound:  Defense  did  not 
complete  an  economic  analysis  until  nearly  3  years  after  the  effort  began, 
and  the  analysis  was  not  updated  even  though  Defense  dramatically 
changed  the  course  of  the  systems  development.  We  are  concerned  that 
Defense  may  repeat  this  experience  unless  it  ensures  that  the  remaining 
migration  system  investment  decisions  are  based  on  sound  planning. 

8.  Defense’s  response  does  not  clearly  specify  why  it  partially  concurred 
rather  than  fully  concurred.  Nevertheless,  it  is  critical  that  in  revising  its 
directives  and  instructions,  dod  require  that  the  psas  document  that  they 
and  the  key  stakeholders  for  their  functional  areas  (including  the  services, 
agencies,  and  major  commands) 

•  have  conducted  thorough  economic  and  risk  analyses  of  alternative 
operational  approaches  (business  case  analyses)  for  accomplishing  the 
mission  of  the  functional  area  and 

•  have  examined  trade-offs  among  the  completing  proposals  and  ranked  the 
alternative  proposals  based  on  mission  impact,  risk,  and  return. 

Undertaking  such  reviews  is  a  sound  business  practice  that  is  consistent 
with  the  planning  and  management  practices  called  for  in  the 
Clinger-Cohen  Act  of  1996. 

9.  We  disagree  with  dod’s  position  that  DODI  7041.3  provides  adequate 
guidance  for  preparing  of  an  economic  analysis  for  an  information  system 
development/modemization  effort.  The  existing  DODI  7041.3  regulation, 
dated  November  7, 1995  ,  is  very  general.  Defense  itself  recognized  that  the 
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regulation  was  insufficient  guidance  for  preparing  an  economic  analysis 
because  it  developed  and  published  an  unofficial  economic  analysis  guide 
to  supplement  the  regulation.  That  unofficial  guide  recommended,  but  did 
not  require,  standard  methods  and  formats  for  preparing  an  economic 
analysis  for  an  information  system  developmenb'modemization  project. 
According  to  a  pa&e  representative,  this  unofficial  guide  is  no  longer 
adequate  because  it  does  not  require  the  degree  of  standardization  in 
methodology  and  analjdic  techniques  needed  to  support  a  portfolio 
management  approach  for  information  technology  investments,  as 
required  by  the  Clinger-Cohen  Act  of  1996.  For  example,  the  guide  does 
not  require  that  returns  on  investment  for  systems 
development/modemization  projects  be  calculated  in  a  standard  manner 
using  a  standard  definition.  This  lack  of  standardization  results  in 
economic  analyses  for  different  systems  that  are  not  comparable  enough 
for  DOD  managers  to  have  a  good  basis  for  deciding  which  systems  offer 
the  highest  payoffs.  Defense  stated  that  in  copjunction  with  its 
implementation  of  a  portfolio  management  approach,  it  plans  to  officially 
publish  appropriate  documents  and  provide  minimum  standards  and 
guidance  for  the  economic  analysis  process.  However,  Defense  has  not 
established  a  time  frame  for  completing  and  pubUshing  the  guidance.  An 
expedited  time  frame  is  needed  so  that  this  effort  can  be  given  adequate 
priority  for  resources. 

10.  Although  dod’s  existing  acquisition  regulations  do  not  require 
independent  review  and  verification  of  alternatives  analyses  for  major 
information  technology  investments,  conducting  independent  reviews  of 
key  decision-making  documents  is  a  good  business  practice  consistent 
with  the  type  of  sound  information  technology  investment  planning 
promoted  by  the  Clinger-Cohen  Act  of  1996.  An  independent  review  of  an 
alternatives  analysis  helps  ensure  that  the  methodology  used  to  conduct 
the  analysis  is  sound  and  that  the  information  is  accurate  and  complete 
and  can  be  relied  on  for  choosing  among  competing  investment  options. 
This  helps  to  ensure  that  the  alternative  analysis  will  serve  its  purpose:  to 
provide  a  reliable,  detailed,  systematic  evaluation  of  alternative  solutions 
in  terms  of  costs  and  accompanying  benefits  so  that  decisionmakers  can 
judge  whether  the  proposed  alternatives  offer  sufficient  military  or 
economic  benefits  to  be  worth  their  cost,  and  to  determine  which 
alternative  is  the  best  approach.  By  not  requiring  independent  review  of 
alternatives  analyses,  Defense  increases  the  risk  of  making  poor  decisions 
to  pursue  particular  options  for  developing  or  acquiring  migration 
systems.  It  also  risks  sending  a  message  throughout  the  Department  that  it 
is  acceptable  to  decided  to  invest  in  major  information  technology 
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projects  without  the  benefit  of  complete  and  accurate  information  on  the 
estimated  costs  and  benefits  of  alternative  investments. 

Where  economic  analyses  are  concerned,  Defense  is  correct  in  stating  that 
its  existing  acquisition  regulations  do  not  require  that  economic  analyses 
for  delegated  systems  be  independently  reviewed  by  pa&e.  Our 
recommendation  has  been  revised  to  enable  verification  by  other  qualified 
reviewers.  However,  our  recommendation  is  stUl  valid  since  our  review 
revealed  that  Defense  does  not  ensure  that  economic  analyses  for 
delegated  systems  are  independently  reviewed  by  any  qualified  reviewer. 

We  identified  seven  major  delegated  migration  systems  for  which 
economic  analyses  had  not  been  prepared  or  updated  and  independently 
reviewed  even  though  substantial  investments  had  already  been  made  in 
them.  The  failure  to  obtain  independent  review  for  these  systems  and  to 
fully  address  concerns  raised  by  the  reviewer  is  unacceptable  since  it 
increases  the  chances  that  inaccurate,  incomplete,  and  unreliable  cost  and 
benefit  information  will  be  used  to  justify  a  costly  system  project. 
Assuming  this  risk  is  hardly  a  wise  choice  given  that  as  a  result.  Defense 
may  end  up  using  scarce  resources  to  build  information  systems  that  offer 
less  payback  than  available  alternatives.  In  addition,  we  believe  that  the 
failure  to  obtain  an  independent  review  of  an  economic  analysis  is  certain 
to  decrease  the  value  of  the  analysis  as  a  management  tool  for  such 
activities  as  Defense’s  formulation  of  the  Program  Objective 
Memorandum,  budget  justifications,  ranking  of  investment  alternatives, 
laying  a  baseline  for  tracking  performance  improvements,  coordination  of 
project  development  schedules  and  costs,  sensitivity  (“what  if’)  analyses, 
and  identification  of  project  components  that  are  unfunded. 

Furthermore,  even  though  dod’s  acquisition  regulations  do  not  require  that 
PA&E  be  the  economic  analysis  reviewer  for  delegated  systems,  because 
PA&E  has  the  necessary  expertise,  it  makes  sense  that  dod  use  pa&e  to 
conduct  these  reviews.  Also,  the  reviews  should  be  more  consistent  if  pa&e 
performs  all  the  economic  analysis  reviews  for  major  systems.  Regardless 
of  whether  pa&e  or  another  qualified  independent  organization  performs 
the  independent  reviews,  it  is  critical  that  an  independent  review  be  made 
of  the  economic  analyses  for  major  information  technology  investments 
cind  that  problems  identified  in  the  analyses  are  addressed  before 
investment  decisions  are  made. 

11.  See  “Agency  Comments  and  Our  Evaluation”  section  in  chapter  4. 
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As  shown  in  figure  II.  1,  Defense’s  migration  system  selection  process  is 
principally  the  responsibihty  of  the  managers  of  Defense  functional  areas 
(such  as  logistics,  finance,  human  resources,  and  health  affairs)  with 
assistance  by  the  technical  staff  who  report  to  and  advise  the  asd  C3I. 


Figure  11.1:  Migration  Seiection  and 
Oversight  Processes 
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Source:  GAO  review  of  Defense  regulations,  memorandums,  related  documents,  and  interviews 
with  Defense  officials. 


The  specific  steps  involved  with  the  process  illustrated  above  are 
explained  in  more  detail  in  the  following  sections. 


Principal  Staff  Assistant 
(PSA)  Selects  the 
Migration  System 


PSAS  are  the  senior  Defense  managers  for  the  functional  areas.  They 
include  Under  Secretaries,  Assistant  Secretaries,  the  Inspector  General, 
the  Director  of  Operational  Test  and  Evaluation,  the  General  Coimsel,  and 
a  number  of  other  top  dod  officials,  psas  are  responsible  for  choosing 
migration  systems  from  their  respective  functional  area  legacy  systems  or 
for  developing  new  systems. 

An  important  first  step  involved  with  deciding  whether  to  migrate  and 
determining  the  migration  system(s)  that  will  best  support  the  functional 
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area  is  assessing  the  functional  area’s  business  processes  and  deciding  if 
changes  are  needed.  For  example,  a  functional  area  may  be  better  off 
outsourcing  a  process  rather  than  attempting  to  bring  improvements 
through  internal  information  system  modifications.  Or,  it  could  decide  that 
it  should  fundamentally  change  its  business  processes  and  acquire 
different  information  systems  or  significantly  modify  existing  systems  to 
better  support  the  new  processes.  In  dod,  the  fimctional  area  goes  about 
making  this  broad  evaluation  of  paths  toward  improvement  and  making  a 
“business  case”  for  the  selected  path  through  the  preparation  of  a 
functional  economic  analysis.  In  most  cases,  functional  areas  have 
concluded  that  migration  is  a  viable  strategy  for  improvement. 

When  it  embarked  on  its  accelerated  migration  strategy  in  1993,  Defense 
allowed  psas  to  tailor  the  functional  economic  analysis,  as  appropriate,  to 
expedite  the  migration  system  selections.  At  the  same  time,  however,  psas 
and  program  managers  who  manage  the  individual  acquisition  projects 
were  told  that  they  should  still  follow  requirements  already  established  for 
developing  and  maintaining  automated  systems  (life-cycle  management 
processes).  For  example,  managers  would  still  need  to  perform  an 
economic  analysis  for  migration  systems  to  establish  a  baseline  life-cycle 
cost  and  benefit  estimate  for  the  projects.^  This  analysis  follows  the 
decision  to  migrate  has  been  made  and  should  be  based  on  an  analysis  of 
alternative  hfe-cycle  costs,  benefits,  and  risks  involved  in  feasible 
approaches  for  the  project. 


'Defense  regulations  require  that  an  economic  analysis  and  an  alternatives  analysis  be  prepared  to 
support  all  major  information  systems  investments  that  meet  the  MAISRC  review  criteria  However, 
for  those  major  systems  that  meet  criteria  for  oversight  by  the  DAB,  Defense  regulations  require  an 
independent  cost  estimate,  rather  than  an  economic  analysis.  For  nonmajor  information  system 
investments  that  do  not  meet  either  MAISRC  or  DAB  review  criteria,  0MB  and  Defense  guidance,  as 
well  as  best  industry  and  government  practices,  recommend  that  an  economic  analysis  and  an 
evaluation  of  alternatives  be  prepared  to  help  decisionmakers  estimate  the  costs  and  benefits  for  the 
project  and  choose  the  most  cost-effective  approach  among  competing  information  technology 
investments. 
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The  ASD  C3I  Approves  the 
Systems  Selected  for 
Migration^ 


After  the  psas  select  the  migration  systems  for  their  functional  activities, 
the  selections  are  to  be  sent  to  the  asd  C3i  for  approval.  In  deciding 
whether  to  approve  the  selections,  the  asd  C3i  reviews  data,  technical,  and 
programmatic  factors.  These  include: 


•  Whether  the  migration  systems  will  lend  themselves  to  data-sharing  within 
their  design.  A  key  step  in  the  process  of  ensuring  this  ability  to  share  data 
is  known  as  data  standardization.  This  involves  reaching  agreement  on 
standard  definitions  that  are  to  be  used  for  the  data  elements’  names, 
meanings,  and  other  characteristics  so  that  all  users  of  the  data  have  a 
common,  shared  understanding  of  it. 

•  Whether  the  migration  systems  currently  conform  to  or  can  evolve  to 
conform  with  Defense’s  technical  standards.  These  technical  standards 
are  a  set  of  “building  codes”  for  constructing  systems  to  ensure  that  they 
will  be  compatible  with  Defense’s  communications  and  computing 
infrastructure  and  technically  interoperable  with  each  other.® 

•  Whether  the  system  selections  are  supported  by  a  ftdl  or  abbreviated 
functional  economic  analysis  or  other  similar  justification. 


The  Defense  Information  Systems  Agency  (disa)  assists  in  the  review  of 
these  factors  and  passes  on  any  concerns  it  has  to  the  asd  C3I. 


Major  Migration  Systems 
Also  Undergo  DOD’s  Major 
Systems  Acquisition 
Review  Process 


In  addition  to  the  management  and  oversight  processes  established 
specifically  for  the  accelerated  migration  strategy,  the  major  migration 
systems  are  also  subject  to  Defense’s  traditional  major  information 
systems  acquisition  oversight  processes  by  either  the  Defense  Acquisition 
Board  (dab)  or  the  Major  Automated  Information  System  Review  CoimcU 
(maisrc).  dab  generally  provides  oversight  for  weapon  systems  while 
MAiSRC  provides  oversight  for  management  information  systems. 


foster  an  Enterprise  Integration  approach  and  augment  the  approval  process  for  initial  migration 
selections,  Defense  established  a  top  management  oversight  structure  to  help  resolve  issues  affecting 
multiple  functional  areas  and  to  promote  cross-functional  planning  and  information-sharing.  This 
structure  consisted  of  two  committees:  (1)  the  Enterprise  Integration  Executive  Board,  which  was 
chaired  by  the  Deputy  Secretary  of  Defense  and  included  the  most  senior  executive  members  in  the 
Department,  and  (2)  the  Enterprise  Integration  Corporate  Management  Council,  which  served  as  a 
supporting  committee  and  was  co-chaired  by  the  Principal  Deputy  Under  Secretary  of  Defense  for 
Acquisition  and  Technology  and  the  ASD  C3I.  However,  these  committees  were  deactivated  in  the 
summer  of  1995 — about  1  year  after  they  were  established — after  the  Deputy  Secretary  decided  they 
were  not  working  as  intended. 

^Defense’s  technical  standards  for  constructing  information  systems  include,  among  others,  the 
Technical  Architecture  Framework  for  Information  Management  (TAFIM)  and  the  Defense 
Information  Infrastructure  Common  Operating  Environment  (DII  COE). 
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Specifically,  dab  reviews  those  systems  that  (1)  are  estimated  to  eventually 
cost  over  $355  milhon  to  research,  develop,  test,  and  evaluate,  (2)  have 
estimated  procurement  costs  of  over  $2,135  bUlion,  or  (3)  are  designated 
for  DAB  review  by  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology,  who  also  chairs  dab.  maisrc  reviews  those  systems  that  (1)  are 
anticipated  to  cost  $30  million  or  more  a  year,  (2)  have  estimated  program 
costs  in  excess  of  $120  million,  (3)  have  estimated  life-cycle  cost  of  more 
than  $360  million,  or  (4)  are  designated  for  maisrc  review  by  the  asd  C3I, 
who  ailso  chairs  the  maisrc.^ 

DAB  and  MAISRC  reviews  involve  assessing  such  matters  as  whether  the 
proposed  system  is  being  developed  in  accordance  with  Defense  policies, 
procedures,  and  regulations;  the  systems’  program  managers  took  steps  to 
minimize  the  cost  of  a  new  system  by  ensuring  full  and  open  competition; 
and  the  program  managers  will  effectively  use  advanced  system  design 
and  software  engineering  technology  to  minimize  software  development 
and  maintenance  costs. 

When  the  dab  or  maisrc  staffs  review  a  migration  system,  they  obtain 
assistance  from  the  Program  Analysis  and  Evaluation  (pa&e)  staff  to 
review  the  system’s  economic  analysis  or  cost  estimate.  This  staff  is 
responsible  for  reviewing  economic  analyses  for  major  information 
systems  and  determining  whether  cost  and  benefit  estimates  are  accurate 
and  complete,  pa&e  analysts  are  recognized  as  dod’s  experts  on  economic 
and  cost  analyses.  Organizationally,  pa&e  reports  to  the  Under  Secretary  of 
Defense  (Comptroller),  maisrc  requests  that  disa  provide  technical  reviews 
of  the  plans  and  other  key  documents  prepared  to  support  the  system’s 
development,  maisrc  also  obtains  assistance  from  other  Defense 
organizations  in  reviewing  the  systems. 

Program  managers  for  the  systems  reviewed  by  maisrc  provide  periodic 
progress  information  to  maisrc  regarding  such  issues  as  cost,  schedule, 
return  on  investment,  and  compliance  with  technical  and  data  standards. 
Generally,  this  information  is  provided  in  written  quarterly  reports,  which 
may  be  tailored  to  meet  the  needs  of  both  the  program  manager  and 
maisrc.  In  addition,  maisrc  conducts  in-process  reviews  of  the  systems  by 
estabhshing  “integrated  product  teams”  that  provide  advice  and  assistance 
to  the  systems’  program  managers  on  various  subjects  as  needed  to 
progress  through  the  acquisition  process. 


■‘These  dollar  amounts  are  in  fiscal  year  1996  constant  dollars  (dollar  amounts  after  they  have  been 
adjusted  for  the  effect  of  inflation). 
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Having  accurate  and  reliable  cost  and  inventory  information  is  critical  for 
decision-making  purposes.  It  helps  Defense  managers  to  determine 
whether  migration  system  investments  will  be  cost-effective  and  whether 
enough  funds  will  be  available  for  developing  and  deploying  the  system. 
However,  Defense’s  migration  cost  information  is  incomplete  and  may  be 
significantly  understated. 

Collecting  cost  information  for  migration  systems  in  itself  is  a  challenging 
task  because  Defense  does  not  maintain  complete  cost  information  on  all 
of  the  migration  systems  in  the  Defense  Integration  Support  Tools  (dist) 
database — its  primary  information  system  tracking  tool.  Thus,  to  report  to 
the  Congress  and  others  on  migration  costs.  Defense  must  turn  to 
numerous  sources.  For  example,  in  providing  information  on  costs  to  the 
Congress  in  April  1996,  Defense  asked  individual  functional  area  managers 
to  provide  total  estimated  cost  data  for  each  of  their  migration  systems  for 
fiscal  years  1995  through  2000. 

Defense  officials  stated  that  cost  data  are  not  maintained  in  dist  because 
functional  area  managers  expressed  concern  that  controls  were  not 
adequate  to  ensure  that  only  authorized  personnel  were  allowed  access  to 
the  data.  They  also  noted  that  Defense  does  not  have  budget  information 
for  all  the  migration  systems  because  it  does  not  separately  budget  or 
collect  information  for  systems  with  annual  cost  of  less  than  $2  million.  In 
fiscal  year  1996,  dod’s  Office  of  the  Under  Secretary  of  Defense 
(Comptroller)  maintained  budget  information  on  the  systems  that  cost 
$2  million  or  more  and  is  currently  working  on  including  all  migration 
systems  and  other  information  system  costs  (for  systems  with  annual 
costs  of  more  than  $2  million)  within  one  database. 


Cost  Information  Is 
Incomplete 


In  December  1996,  Defense  provided  us  vnth  cost  information  on  the 
migration  effort,  but  these  data  provided  costs  for  only  242  of  the  363 
migration  systems.  We  did  not  include  costs  in  this  report  for  27  of  242 
systems  for  which  dod  collected  costs  because  the  systems  were 
classified.  Defense  did  not  provide  us  costs  for  the  remaining  121  systems 
because  the  Office  of  the  asd  C3i  had  not  collected  costs  for  these 
systems. 

When  we  reviewed  the  cost  data  reported  for  the  clinical  health,  civihan 
personnel,  and  transportation  areas,  as  part  of  our  review  of  migration 
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systems’  cost  reported  to  the  Congress  in  April  1996/  we  identified  three 
reasons  why  the  functional  areas  did  not  report  costs  for  some  systems. 
First,  for  some  of  the  systems,  these  costs  were  not  available  at  the  time 
the  report  was  made  to  the  Congress.  Second,  for  other  systems,  the 
functional  area  managers  were  not  aware  that  the  report  was  to  include 
interim  migration  systems.  Third,  for  some  systems,  functional  area 
approval  as  a  migration  system  may  not  have  been  completed  at  the  time 
cost  data  were  gathered  for  the  report  to  Congress.  As  table  lll.l 
illustrates,  the  costs  not  reported  represented  a  substantial  portion  of  the 
total  being  spent  on  migration  for  the  three  functional  areas. 


Table  Costs  Reported  to  the 

Congress  Compared  to  Costs 

Dollars  in  millions 

Reported  to  GAO  for  Clinical  Health, 
Civilian  Personnel,  and  Transportation 

Functional  area 

Total  costs 
reported 
to  the 
Congress 

Total  costs 
reported 
to  GAO 

Difference 
(amount  not 
reported  to  the 
Congress) 

Clinical 

Health 

$1,014.0 

$2,451.3 

$1,437.3 

Civilian 

Personnel 

67.4 

123.9 

56.5 

Transportation 

1,005.2 

1,116.6 

111.4 

Total 

$2,086.6 

$3,691.8 

$1,605.2 

Source:  GAO’s  analysis  of  information  obtained  from  the  April  1996  Report  to  the  Congress, 
Defense  functional  area  managers,  and  the  ASD  C3I. 


Significant  Costs  Not 
Tracked 


We  also  found  that  Defense  does  not  account  for  some  very  important 
costs  related  to  developing,  deploying,  and  maintaining  migration  systems 
both  before  and  after  a  project  has  been  initiated.  It  is  extremely  important 
to  have  accurate  estimates  up  front  on  direct,  indirect,  and  support  costs 
associated  with  a  system  development  or  acquisition  project  and  to  be 
able  to  track  these  costs  throughout  the  Ufe-cycle.  At  the  beginning  of  a 
system  development  project,  a  complete  cost  picture  provides  an 
organization  with  a  realistic  picture  of  what  it  will  take  to  develop,  deploy, 
and  maintain  a  system  over  its  life-cycle.  It  also  enables  an  organization  to 
make  sound  trade-off  decisions  among  competing  investments. 

Continually  tracking  direct,  indirect,  and  support  costs  helps  managers 
account  for  their  past  activities,  manage  current  operations,  and  assess 
progress  toward  planned  objectives. 


‘As  stated  in  chapter  1,  this  report  was  made  pursuant  to  a  requirement  in  Section  366  of  the  National 
Defense  Authorization  Act  for  Fiscal  Year  1996. 
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As  noted  in  this  report,  when  we  conducted  detailed  reviews  of  selected 
finance  and  logistics  migration  efforts,  we  found  that  dod  excluded 
significant  categories  of  costs  from  its  initial  life-cycle  cost  estimates. 
Furthermore,  our  work  in  the  clinical  health,  civilian  personnel,  and 
transportation  fxmctional  areas  showed  that  dod  also  does  not  capture  and 
report  the  total  actual  costs  for  its  systems  as  they  are  developed, 
deployed,  and  maintained.  Instead,  dod  decisionmakers  employ  cost 
accounting  systems  that  omit  relevant  project  costs,  such  as  those 
associated  vsdth  project  management  and  oversight.  The  transportation 
functional  area,  for  example,  has  an  office  designated  to  oversee  the 
development  and  deployment  of  its  migration  systems.  But,  when 
accoimting  for  costs  for  the  systems,  dod  does  not  factor  in  the  cost  to 
maintain  this  oversight  responsibility.  As  a  result,  dod  cannot  reliably 
report  migration  system  projects’  actual  costs  and  compare  them  against 
established  baselines,  and  it  cannot  reliably  use  information  relating  to 
projects’  actual  costs  to  improve  future  cost  estimating  efforts. 

Beginning  October  1,  1996,  a  new  federal  accounting  standard  requires  the 
Department  of  Defense  and  other  federal  agencies  to  begin  accounting  for 
all  identifiable  direct,  indirect,  and  support  costs  of  its  outputs.  This  would 
provide  a  mechanism  for  developing  total  costs  of  capital  items,  such  as 
information  systems.  This  requirement  is  defined  in  the  Statement  of 
Federal  Financial  Accounting  Standards,  (sffas)  No.  4,  Managerial  Cost 
Accoimting  Concepts  and  Standards  for  the  Federal  Government. 
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There  are  a  number  of  data  integrity  problems  associated  with  the  dist 
database,  which  is  used  to  keep  information  on  Defense’s  migration 
systems.  These  problems  directly  affect  Defense’s  ability  to  accurately 
determine  how  many  legacy  systems  were  terminated  and  how  many  are 
scheduled  for  termination.  Defense  officials  acknowledge  that  the  dist 
database  is  incomplete  and  inaccurate,  but  stated  they  used  it  to  generate 
reports  to  the  Congress  because  it  was  the  only  departmentwide  database 
containing  schedule  and  other  descriptive  data  on  all  of  Defense’s 
migration  and  legacy  systems. 

When  we  obtained  a  copy  of  the  dist  database  for  analysis  in  April  1996, 
Defense’s  own  analysis  of  the  dist  database  confirmed  that  it  had 
incomplete  data  in  several  categories  of  key  information.  A  more  recent 
Defense  analysis  of  migration  systems  showed  that  dist  continues  to  have 
high  levels  of  incomplete  data.  Table  IV.  1  illustrates  how  information  was 
incomplete  in  several  categories  of  information. 


Table  tV.1:  Levels  of  Incompleteness  in 
DIST  in  Aprii  1996  and  February  1997 


Data  category 

Percent  of  systems 
with  incomplete  data 

April  February 
1996  1997 

Interfaces  with  other  systems 

88 

55 

Computer  installations  where  the  system  operated 

79 

77 

Computer  hardware  on  which  the  system  operated 

75 

68 

System  software 

66 

61 

Organization  responsible  for  the  system 

32 

26 

Source:  GAO  analysis  of  DIST  Data  Status  Working  Papers. 


Our  analysis  supports  Defense’s  concerns  as  still  relevant  and  identified 
the  following  additional  data  integrity  problems. 

•  Large  numbers  of  migration  system  implementation  dates  (220,  or 
61  percent)  of  the  migration  systems’  implementation  dates  and  legacy 
system  termination  dates  (629,  or  32  percent)  shown  in  dist  were  in  the 
year  2010.  According  to  system  documentation  and  our  discussions  with 
officials  responsible  to  operate  dist,  the  presence  of  the  year  2010  for 
these  dates  indicates  the  system  owners  responsible  for  the  migration 
systems  did  not  provide  a  date,  resulting  in  the  dist  system  automatically 
inserting  the  default  date  of  2010. 
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•  As  figure  IV.  1  illustrates,  when  we  compared  the  dist  schedule  data  for 
clinical  health,  civilian  personnel,  and  transportation  to  the  actual 
schedule  maintained  for  these  areas,  we  found  that  incorrect  data  for 
migration  implementation  and  legacy  system  termination  outweighed 
correct  data. 


Figure  IV.1:  Number  of  Incorrect  and 
Correct  Migration  and  Legacy  System 
Dates  for  Clinical  Health,  Civilian 
Personnel,  and  Transportation 


Migration  system  Legacy  system 

impiementation  termination  dates 

dates 


Correct 


□  Incorrect 


Sources:  GAO’s  analysis  of  information  obtained  from  the  DIST,  April  1996  Report  to  the 
Congress,  Defense  functional  area  managers,  and  ASD  C3I. 


Functional  area  managers  in  these  three  areas  identified  additional 
inaccuracies.  For  example,  managers  in  the  transportation  area  stated  that 
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the  area  had  implemented  six  migration  systems  rather  than  none  as 
indicated  in  dist.  For  the  three  functional  areas,  dist  showed  that  92  legacy 
systems  were  terminated  by  April  1996,  while  functional  managers  told  us 
that  only  43  had  actually  been  terminated.  Also,  for  the  three  functional 
areas,  dist  showed  that  53  legacy  systems  were  scheduled  for  future 
termination,  but  functional  managers  told  us  that  91  were  slated  for  future 
termination. 

We  also  learned  that  dist  has  other  problems  that  hamper  its  use.  One 
problem  is  that  Defense  has  not  ensured  that  the  data  definitions  and 
formats  used  in  dist  are  ftilly  compatible  with  data  maintained  in  other 
Defense  information  systems  that  track  and  report  on  systems.  While  dod 
is  attempting  to  address  this  issue,  disa  provided  us  information  showing 
that  only  66  of  the  218  data  elements  contained  in  the  dist  have  been 
approved  as  standard  data  elements  in  the  Defense  Data  Dictionary 
System.  Without  standard  definitions  and  data  formats,  data  cannot  be 
easily  transferred  to  dist  from  other  systems  that  may  be  used  by  psas, 
program  managers,  and  other  decisionmakers. 

In  addition.  Defense  assigns  unique  numbers  to  each  system  in  dist,  but 
these  numbers  cire  not  universally  used  throughout  the  Department.  The 
different  number  schemes  make  it  more  difficult  to  facilitate  the  exchange 
of  information  between  these  systems.  Use  of  the  same  identifying  number 
for  each  system  throughout  dod  is  also  important  because  Defense  often 
changes  the  names  of  its  information  systems  over  their  life.  Without 
consistent  identification  numbers,  tracking  an  information  system  in  the 
face  of  such  name  changes  is  more  arduous. 

Defense  recognizes  that  it  needs  better  information  to  deal  with  migration 
and  other  technology  issues  for  which  dist  is  reUed  on,  such  as  the  Year 
2000  problem.^  In  November  1996,  the  Under  Secretary  for  Defense 
(Comptroller)  and  the  asd  C3i  issued  a  joint  memorandum  to  senior 
Defense  managers  stating  that  dist  was  the  backbone  tool  for  managing 
the  Department’s  information  technology  investment  and  that  to  be 
effective,  it  must  contain  accurate  data  on  all  the  Department’s 
information  systems.  The  memorandum  also  stated  that  registration  of 
information  systems  in  the  dist  was  mandatory.  However,  the  minimum 
set  of  data  that  is  required  under  this  joint  memorandum  focused  on 
information  needed  to  address  Year  2000  issues  and  did  not  include 


^The  Year  2000  problem  refers  to  the  possibility  that  many  computer  systems  will  fail  on  New  Year’s 
morning  in  the  year  2000  because  they  were  not  designed  to  accommodate  the  change  of  date  to  the 
new  millennium. 
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Appendix  IV 

Detailed  Analysis  of  Database  Used  to 
Collect  Migration  Information 


critical  information  on  migration  implementation  dates,  cost,  savings,  and 
performance  measures.^ 


‘See  chapter  4,  footnote  1,  for  recent  Defense  developments  regarding  DIST. 
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Appendix  V _ 

Major  Contributors  to  This  Report 


Accounting  and 
Information 
Management  Division, 
Washington,  D.C. 


Cristina  T.  Chaplain,  Communications  Analyst 
Danny  R.  Latta,  Assistant  Director,  Policies  and  Issues 
Harold  J.  Brumm,  Jr.,  Economist 


Kansas  City  Field 
Office 


George  L.  Jones,  Evaluator-in-Charge 
David  R.  Solenberger,  Senior  Evaluator 
Karen  S.  Sifford,  Staff  Evaluator 
John  G.  Snavely,  Staff  Evaluator 
Michael  W.  Buell,  Staff  Evaluator 
Karl  G.  Neybert,  Staff  Evaluator 


I 

I 
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Related  GAO  Products 


High-Risk  Series:  Information  Management  and  Technology  (gao/hr-97-9, 
February  1997). 

Air  Traffic  Control:  Improved  Cost  Information  Needed  to  Make  Billion 
Dollar  Modernization  Investment  Decisions  (gao/aimd-97-20,  January  22, 
1997). 

Defense  Communications:  Performance  Measures  Needed  to  Ensure  DISN 
Program  Success  (gao/aimd-97-9,  November  27,  1996). 

Defense  irm:  Strategy  Needed  for  Logistics  Information  Technology 
Improvement  Efforts  (gao/aimd-97-6,  November  14, 1996). 

Information  Technology  Investment:  Agencies  Can  Improve  Performance, 
Reduce  Costs,  and  Minimize  Risks  (gao/aimd-96-64,  September  30, 1996). 

POD  Accoimting  Systems:  Efforts  to  Improve  System  for  Navy  Need  Overall 
Structure  (gao/aimd-96-99,  September  30, 1996). 

Defense  irm:  Critical  Risks  Facing  New  Materiel  Management  Strategy 
(gao/aimd-96-109,  September  6, 1996). 

Defense  Transportation:  Migration  Systems  Selected  Without  Adequate 
Analysis  (gao/aimd-96-8i,  August  29, 1996). 


Information  Management  Reform:  Effective  Implementation  Is  Essential 
for  Improving  Federal  Performance  (gao/t-aimd-96-132,  July  17,  1996). 

Acquisition  Reform:  Regulatory  Implementation  of  the  Federal  Acquisition 
Streamlining  Act  of  1994  (gao/nsiad-96-139,  June  28, 1996). 

Medical  ADP  Systems:  Defense  Achieves  Worldwide  Deployment  of 
Composite  Health  Care  System  (gao/aimd-96-39,  April  5, 1996). 

Best  Management  Practices:  Reengineering  the  Air  Force’s  Logistics 
System  Can  Yield  Substantial  Savings  (gao/nsiad-96-5,  February  21, 1996). 

Inventory  Management:  pod  Can  Build  on  Progress  in  Using  Best  Practices 
to  Achieve  Substantial  Savings  (gao/nsiap-95-142,  August  4, 1995). 

Information  Technology  Investment:  A  Govemmentwide  Overview 
(gao/aimp-95-208,  July  31, 1995). 
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Related  GAO  Products 


Defense  Management:  Selection  of  Depot  Maintenance  Standard  System 
Not  Based  on  Sufficient  Analyses  (gao/aimd-95-iio,  July  13, 1995). 

Defense  Communications:  Management  Problems  Jeopardize  DISN 
Implementation  (gao/aimd-96-136,  July  13, 1995). 


Best  Practices  Methodology:  A  New  Approach  for  Improving  Government 
Operations  (gao/nsiad-95-154.  May  1995). 

Government  Reform:  Goal-Setting  and  Performance  (gao/aimd/ggd-95-130R, 
March  27, 1995). 

Defense  Infrastructure:  Enhancing  Performance  Through  Better  Business 
Practices  (gao/t-nsiad/aimd-95-126,  March  23, 1995). 

Paperwork  Reduction  Act:  Reauthorization  Can  Strengthen  Government’s 
Management  of  Information  and  Technology  (gao/t-aimd/ggd-95-80, 

February  7,  1995). 

Government  Reform:  Using  Reengineering  and  Technology  to  Improve 
Government  Performance  (gaoa’-ocg-95-2,  February  2, 1995). 

High-Risk  Series:  An  Overview  (gao/hr-95-i,  February  1995). 

Reengineering:  Opportunities  to  Improve  (gao/aimd-95-67R,  January  6, 1995). 

Reengineering  Organizations:  Results  of  a  gap  Symposium  (gao/nsiad-95-34, 
December  13,  1994). 

Defense  Management:  Impediments  Jeopardize  Logistics  Corporate 
Information  Management  (gao/nsiad-95-28,  October  21, 1994). 


pod’s  cals  Initiative  (gao/aimd-94-i97R,  September  30, 1994). 

Defense  Management  Initiatives:  Limited  Progress  in  Implementing 
Management  Initiatives  (gao/t-aimd-94-106,  April  14, 1994). 

Financial  Management:  Financial  Control  and  System  Weaknesses 
Continue  to  Waste  pod  Resources  and  Undermine  Operations 
(gao/t-aimd/nsiad-94-154,  April  12, 1994). 
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Related  GAO  Products 


Defense  Management:  Stronger  Support  Needed  for  Corporate 
Information  Management  Initiative  to  Succeed  (gao/aimd/nsiad-94-ioi, 

April  12, 1994). 

Defense  irm:  Management  Commitment  Needed  to  Achieve  Defense  Data 
Administration  Goals  (gao/aimd-94-14,  January  21, 1994). 

Defense  irm:  Business  Strategy  Needed  for  Electronic  Data  Interchange 
Program  (gao/aimd-94-17,  December  9, 1993). 


Financial  Mamagement:  Reliability  of  Weapon  System  Cost  Reports  Is 
Highly  Questionable  (gao/aimd-94-io,  October  28, 1993). 


Defense:  Health  Care  (gao/imtec-93-29R,  Jime  3, 1993). 

Mission-Critical  Systems:  Defense  Attempting  to  Address  Major  Software 
Challenges  (gao/imtec-93-13,  December  24, 1992). 

Defense  adp:  Corporate  Information  Management  Must  Overcome  Major 
Problems  (gao/imtec-92-77,  September  14, 1992). 

Air  Force  adp:  Status  of  Logistics  Modernization  Projects  and  cim  Impacts 
(gao/imtec-92-66,  July  30, 1992). 

Defense  Logistics:  dod  Initiatives  Related  to  Cutting  Costs 
(gao/t-nsiad-92-24,  March  26, 1992). 

Defense  adp:  Lessons  Learned  From  Development  of  Defense  Distribution 
System  (gao/imtec-92-25,  March  20, 1992). 


Automated  Information  Systems:  Defense  Should  Stop  Further 
Development  of  Duplicative  Recruiting  Systems  (gao/imtec-92-i5, 
February  27, 1992). 

Defense  adp:  A  Coordinated  Strategy  Is  Needed  to  Implement  the  cals 
Initiative  (gao/imtec-91-54,  September  13, 1991). 

Defense  Distribution  System  Illustrates  Problenas  Facing  Corporate 
Information  Management  Interim  Standard  Systems  (gao/t-imtec-91-12, 
April  23, 1991). 
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Related  GAO  Products 


Defense  adp:  Corporate  Information  Management  Initiative  Faces 
Significant  Challenges  (gao/imtec-91-35,  April  22, 1991). 


Defense  adp:  Corporate  Information  Management  Savings  Estimates  Are 
Not  Supported  (gao/imtec-91-18,  February  22, 1991). 

Army  Automation:  Decisions  Needed  on  SIDPERS-3  Before  Further 
Development  (gao/imtec-90-66,  September  5, 1990). 


Automated  Information  Systems:  Defense’s  Oversight  Process  Should  Be 
Improved  (gao/imtec-90-36,  April  16, 1990). 

Schedule  Delays  and  Cost  Overruns  Plague  pod  Automated  Information 
Systems  (gaoa'-imtec-89-8.  May  18, 1989). 

Automated  Information  Systems:  Schedule  Delays  and  Cost  Overruns 
Plague  POD  Systems  (gao/imtec-89-36.  May  10, 1989). 


(611355) 
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